# Noam Shazeer and Jack Rae: Scaling Test-time Compute, Reactions to Ilya & AGI Markdown copy of https://realalpha.app/v/atMRWzgHEGg/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy. Noam Shazeer and Jack Rae argue that test-time compute can raise model capability, but reaching AGI also requires agents that act in complex environments. - Show: Unsupervised Learning - Hosts: [Jacob Effron](https://realalpha.app/people/jacob-effron/) - Guest: [Noam Shazeer](https://realalpha.app/builders/noam-shazeer/) · OpenAI; co-author of the Transformer - Guest: [Jack Rae](https://realalpha.app/builders/jack-rae/) · Research Director, DeepMind - Published: Mar 17, 2025 - Length: 1 h 10 m ## The best minutes - [19:34](https://realalpha.app/v/atMRWzgHEGg/?t=1174) [Inference is cheap enough to justify much more thinking](https://realalpha.app/s/inference-is-cheap-enough-to-justify-much-more-thinking/) (6 min) - [25:29](https://realalpha.app/v/atMRWzgHEGg/?t=1529) [AI math progress depends on asking better questions](https://realalpha.app/s/ai-math-progress-depends-on-asking-better-questions/) (6 min) - [31:48](https://realalpha.app/v/atMRWzgHEGg/?t=1908) [Research teams need both bottom-up and top-down bets](https://realalpha.app/s/research-teams-need-both-bottom-up-and-top-down-bets/) (8 min) ## Also worth watching - [9:39](https://realalpha.app/v/atMRWzgHEGg/?t=579) [Qualitative tasks need broad reward criteria](https://realalpha.app/s/qualitative-tasks-need-broad-reward-criteria/) (2 min) - [15:57](https://realalpha.app/v/atMRWzgHEGg/?t=957) [Agent research depends on choosing the right environment](https://realalpha.app/s/agent-research-depends-on-choosing-the-right-environment/) (4 min) - [42:03](https://realalpha.app/v/atMRWzgHEGg/?t=2523) [Use one model unless specialization earns its cost](https://realalpha.app/s/use-one-model-unless-specialization-earns-its-cost/) (2 min) - [1:06:30](https://realalpha.app/v/atMRWzgHEGg/?t=3990) [Inference-heavy AI can use more distributed compute](https://realalpha.app/s/inference-heavy-ai-can-use-more-distributed-compute/) (3 min) - [58:15](https://realalpha.app/v/atMRWzgHEGg/?t=3495) [Hallucinations made entertainment a natural early use](https://realalpha.app/s/hallucinations-made-entertainment-a-natural-early-use/) (2 min) Skip the extended discussion of AI in family life, human meaning, and social risks. ## Full analysis: Noam Shazeer says inference gets like over a million tokens per dollar - Shazeer says efficient inference could yield like over a million tokens per dollar. - That supports more thinking, but Rae says test-time compute alone will not take models to AGI. - Teams should judge it by verified work, latency, and failure costs, not token prices alone. Its 7 sections, what it means for you and what to watch are on [the page](https://realalpha.app/v/atMRWzgHEGg/#analysis), free with an account: - [20:09](https://realalpha.app/v/atMRWzgHEGg/?t=1209) Inference prices leave room for more thinking - [12:38](https://realalpha.app/v/atMRWzgHEGg/?t=758) Consumers may tolerate added latency, but evidence is thin - [10:16](https://realalpha.app/v/atMRWzgHEGg/?t=616) Broad reward criteria make evaluation design central - [19:21](https://realalpha.app/v/atMRWzgHEGg/?t=1161) Agent environments shape what models can learn - [31:42](https://realalpha.app/v/atMRWzgHEGg/?t=1902) Useful math depends on asking better questions - [23:32](https://realalpha.app/v/atMRWzgHEGg/?t=1412) Rae says test-time compute will not reach AGI alone - [36:57](https://realalpha.app/v/atMRWzgHEGg/?t=2217) Research needs bottom-up exploration and top-down bets ## Summary Shazeer co-invented the Transformer and mixture of experts and works on Gemini. Rae is a research director at DeepMind and one of the leads behind Gemini Flash Thinking. Shazeer sees test-time compute as an inexpensive way to spend more inference on harder problems, while Rae says it will not by itself deliver AGI and that models must also learn to act in complex environments. They argue that progress depends on better evaluations, feedback and agentic environments, and Rae sees deeper reasoning as a route to greater data efficiency and useful research. They describe AI research as experimental and say both bottom-up exploration and top-down bets have value. They also discuss open-source competition, AI’s effects on education and work, safety, and AI companions. ### Minute by minute - [2:03](https://realalpha.app/v/atMRWzgHEGg/?t=123) **Thinking can improve creative work.** Rae says Gemini’s first test-time-compute effort focused on math and code, but thinking also improved creative writing by helping models explore and revise ideas. Shazeer says difficult benchmarks help separate reasoning gains from performance boosts due to more parameters or memorization, while Rae notes that evaluations can quickly saturate or leak. - [6:44](https://realalpha.app/v/atMRWzgHEGg/?t=404) **AI can accelerate AI development.** Shazeer calls using AI to make AI researchers more productive the most important reinforcement loop, alongside user-feedback data flywheels. Rae says Google’s structured codebase and tools already let AI surface fixes and reviews, and he expects agents to handle more open-ended coding work. - [9:39](https://realalpha.app/v/atMRWzgHEGg/?t=579) **Open-ended work needs better rewards.** Shazeer says less-verifiable fields need better ways to verify results or more human feedback. Rae says models can learn broad qualitative criteria, such as style or whether work is interesting, and be trained against those rewards with reinforcement learning. - [11:51](https://realalpha.app/v/atMRWzgHEGg/?t=711) **Thinking and vision broaden use.** Rae says the Gemini app adds thinking to tools including Maps and Search, and users may tolerate a few seconds of latency for better answers. He recounts his mother reading the model’s thoughts after asking about the meaning of life. On multimodal work, Rae points to visual reasoning and Project Mariner’s browser actions, while Shazeer says image generation has fewer training examples than text. - [15:57](https://realalpha.app/v/atMRWzgHEGg/?t=957) **Reliable agents need useful environments.** Shazeer says making models smarter helps, but agents need general solutions because users will find uses developers cannot anticipate. Rae says research also needs well-designed environments, such as a web interface or codebase, and that engineering agentic training is a substantial part of the challenge. - [19:34](https://realalpha.app/v/atMRWzgHEGg/?t=1174) **Inference leaves room for more compute.** Shazeer says efficient inference can deliver over a million tokens per dollar, compared with about 10,000 tokens per dollar for reading a paperback, leaving room to spend more compute when it improves results. Rae says test-time compute alone will not reach AGI; models also need to act in complex environments. - [23:45](https://realalpha.app/v/atMRWzgHEGg/?t=1425) **Deep reasoning could support discovery.** Rae says he wants models to think deeply enough to derive useful knowledge, not simply spend longer on each problem; reinforcement learning with test-time compute is already helping models learn more from available data. He sees a path from math evaluations toward useful scientific work, but says asking novel questions remains difficult. Shazeer argues that connecting known ideas can accelerate science and that math can grow from very little data. - [31:48](https://realalpha.app/v/atMRWzgHEGg/?t=1908) **Research needs experiments and mixed bets.** Shazeer compares AI research to experimental alchemy: researchers form hypotheses, test them and often struggle to assign credit. On resource allocation, he describes Google Brain as more bottom-up and DeepMind as more top-down. Rae says both approaches matter: concentrated bets need vision, but bottom-up work can reveal unexpected breakthroughs. - [39:33](https://realalpha.app/v/atMRWzgHEGg/?t=2373) **General models can span domains.** Both say choosing large language models was a good bet and discuss the move toward vision and native image generation. Shazeer argues that high-value areas such as healthcare do not need separate task-specific models unless cost justifies them. Rae says specialization makes sense if negative transfer appears, while positive transfer favors one general model. - [44:03](https://realalpha.app/v/atMRWzgHEGg/?t=2643) **Open models are closing the gap.** Rae says the speed at which labs adopted test-time compute surprised him: models explored the idea and appeared within months. Both say open-source models are staying close to frontier systems; Rae praises Gemma 3, while Shazeer says the time gap may be small even if a quality gap remains. - [49:04](https://realalpha.app/v/atMRWzgHEGg/?t=2944) **AI could reshape education and purpose.** Rae says his son, under supervision, uses Gemini to identify plants and lizards from photos, and he expects AI to make personalized education more powerful. Shazeer says human labor may not be materially necessary in the future, so people should do meaningful work now and consider where they will find purpose. - [55:39](https://realalpha.app/v/atMRWzgHEGg/?t=3339) **Safety risks need active work.** Rae says he is moderately concerned that more-intelligent systems may not remain predictable or useful, and he flags employment shifts and unintended consequences. Shazeer says he is not afraid but agrees that teams need to work on safety. - [58:05](https://realalpha.app/v/atMRWzgHEGg/?t=3485) **AI companions may serve users differently.** Shazeer says he left Google to start Character because he wanted an application that let people interact with LLMs and discover useful applications. He says entertainment worked partly because hallucinations could be a feature, and he expects people to use AI for both human-like interaction and practical help while still valuing human relationships. - [1:03:00](https://realalpha.app/v/atMRWzgHEGg/?t=3780) **Evaluation and infrastructure still matter.** Rae calls the ARC-AGI evaluation overhyped because progress on specific puzzles may not advance useful AGI, while Shazeer says AGI and coding are underhyped. Rae sees value in agentic applications beyond coding, while Shazeer would focus on engineering agents that help develop AI. Rae says inference-heavy systems could distribute actors across data centers; Shazeer adds that transformer inference can become memory-bound, leaving work for model architecture and hardware. ## Transcript Unsupervised Learning, hosted by Jacob Effron, with Noam Shazeer and Jack Rae. Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there. [0:38](https://realalpha.app/v/atMRWzgHEGg/?t=38) Noam Shazeer and Jack Ray really need no introduction. The two are at the forefront of Google's Gemini LLM efforts and have been involved in some of the most important discoveries in AI in the last decade. Noam as one of the co-inventors of the Transformer and mixture of experts. Jack, a key part of many DeepMind breakthroughs. It's a real privilege of the job to get to sit with these two and ask them literally every top-of-mind question in AI today. We talked about how far test-time compute will get us and the spaces where it will and won't work. We talked about how the infrastructure needs will be different for test-time compute versus the large kind of pre-training paradigm. [1:09](https://realalpha.app/v/atMRWzgHEGg/?t=69) We also hit on the impressive pace at which open source models have caught up with closed source peers and their reactions to DeepSeq. We talked about their kind of reception and reaction to Ilya saying that this test-time compute paradigm won't get us all the way to AGI, as well as Jan LeCun saying this current generation of models can't actually have any novel thoughts. We talked about what it actually looks like to do cutting-edge AI research today and what their day-to-days look like, as well as the future model milestones that actually matter to them. And then we also got Gnome's reflections on character as well as both of their responses to what, you know, AGI means for the role of humanity. [1:41](https://realalpha.app/v/atMRWzgHEGg/?t=101) I think folks are going to love this. It was really just a pleasure to get to speak with both Noam and Jack. Without further ado, here they are. Well, Noam and Jack, thanks so much for coming on the podcast. Oh, thank you. Are we sitting in the very office that the Transformer was invented? No, this is a new building. We were in 1965 Charleston, I think. Probably about half a mile. Half a mile. All right, so it's in the air. Pretty close, pretty close. Well, many things to dive into today. I mean, obviously want to start with some of the latest Gemini 2.0 models and obviously all the work you've been doing around test time compute and Gemini 2.0 flash thinking. [2:16](https://realalpha.app/v/atMRWzgHEGg/?t=136) I guess just at the highest level for our listeners, how do you characterize where these models work today, where they don't work as well? And as you were kind of experimenting with them, what surprised you most about those results? One surprising thing is kind of when we started the particular concerted effort to build a a lot of kind of research into test time compute, into Gemini, and then think about shipping it, is that we were really kind of focused on starting out with reasoning tasks. So math and code were like big areas of focus. And it wasn't really clear, you know, whilst we're kind of sprinting in that kind of domain, we obviously want to broaden it naturally over time, but it wasn't really clear like how that would work. [2:58](https://realalpha.app/v/atMRWzgHEGg/?t=178) would there be any kind of sense of generalization? Would thinking be useful beyond those reasoning tasks if we're just concentrating on those as researchers? And I think it was pretty fun to see one of the early kind of models that we felt like had been trained to try and match the style of Gemini Flash. So it had been trained with thinking, but then it also undergone some kind of training to actually be just like generally a nice style, a nice like model to talk with. that it was actually very fun seeing thinking, interact, and improve creative tasks as well. You could ask the document to compose an essay on a particular topic. [3:37](https://realalpha.app/v/atMRWzgHEGg/?t=217) And actually, A, the thought content was very fun to read. And it would go through various different ideas. And then it would go through revisions of the idea or things that it should cut. And that was kind of fun. And then also the output felt really nice. So that was one thing that kind of surprised me. Any surprise for you, Noam? Well, yeah, I mean, in general, like, I'm all for, like, generality, like, let's train something that's great at everything. It is important, you know, and I was skeptical at first of, like, okay, this intense focus on things like math. But it is very important to have good benchmarks that are going to encourage you to be able to reason about difficult tasks, because a lot of things will drop perplexity, like add more parameters to the model and memorize more. [4:32](https://realalpha.app/v/atMRWzgHEGg/?t=272) So it's nice to have the evals that can distinguish. better, you know, some of the, you know, more difficult problems. I mean, what evals are even meaningful to you at this point? I mean, obviously, I feel like people are trying to hill climb the same set of evals that feel increasingly less relevant to day-to-day work. What do you guys, like, do when you're testing these models, or, you know, how do you vibe check them? It feels like we keep landing on an eval, and we're like, oh, actually, we overlook this eval, even if it's in math. It's like, okay, we've done a bunch of, like, math evals, but maybe, like, I don't know, Putnam answers only, AIME, they're still considered challenging. [5:09](https://realalpha.app/v/atMRWzgHEGg/?t=309) And then it's like, done, okay. They're completely saturated and we really don't care and they're small. And we almost think, why did we ever even work on them? And it's easy to forget. That was so easy, but we thought six months ago. Yeah, a couple of months ago, it was considered really hard, maybe too hard for the model. And then it only snaps to being trivial. So right now, I do feel like there's always been a lot of concerted effort within DeepMind and within like Google as a whole to develop useful evals. But it's kind of very nice to see also this is like a shared responsibility across like many different AI labs and even like scale has like started really stepping up and developing like very challenging evals. [5:49](https://realalpha.app/v/atMRWzgHEGg/?t=349) I think Chalent Humanity's last exam was a dangerous, you know, if every six months we think it wasn't too hard, it's a dangerous title. The really, really last exam. Yeah, exactly. But the whole thing is... V73 last exam. It's very, very challenging in general because you know evals get leaked you know like once people start talking about the evals then there's all this text out there about the evals and um and you know they're they're no good at anymore because everyone knows the problems and you know all the models will uh know the problems unless you're very very careful so there's you know i think there's still a lot of work that It goes on into having evals that are private. [6:32](https://realalpha.app/v/atMRWzgHEGg/?t=392) Are there a set of milestones that are meaningful to you? It's like, hey, when Gemini 3.0 can do X, that's a really exciting milestone, whether it's an eval or just something that you've tried with these models and they can't quite do yet. I'd say when Gemini 3.0 writes Gemini 4.0. Or I should say Gemini X writes Gemini X plus one. Yeah, I think these reinforcement loops are are probably the most important thing to pay attention to and you know there are there are several reinforcement loops going on you know what the one I just mentioned is probably the most important one that we can actually use the AI we're building as as a tool to make ourselves more productive at at building AI but then you know that then there are other reinforcement loops around, you know, data flywheels, like, you know, you get, you have people use these models and provide feedback and make them better at the things that people care about. I think we'll see a huge acceleration from that. And then there's just the global excitement and funding flywheel, which is, I think, also seems to be kicking up in the last few years yes certainly so um you know to your point i guess of having you kind of uh armed with like a thousand ai engineers by your side uh making you even more productive like where are we [7:56](https://realalpha.app/v/atMRWzgHEGg/?t=476) in you know do you have the equivalent of a 0.1 of an fte today of using some of these uh gemini models alongside your research yeah i think one benefit we have at google is like uh we work in a very like structured mono repo so uh and we have a lot of like amazing tooling around like contributing to the code base already so there's like lots of angles where ai is kind of uh like being pulled in as tooling for our own development like uh i i think jeff quoted the statistical i don't know but like the number of what we could call like pull requests that like 25 right yeah like useful like a just like ai like bug fixes or uh code reviews attached that's already like that just It gets pulled in one day. I notice it there. I'm like, oh, that's cool. I can now do a lot of apply fixes that it already spots. [8:46](https://realalpha.app/v/atMRWzgHEGg/?t=526) But that's just one element where we're already pulling in AI towards our own coding development. I think we're incredibly excited about agentic coding is just definitely very important. and trying to get the model to be able to tackle more open-ended and difficult tasks is definitely something we're very excited about. And I think it's just like, in some ways, it's a lot easier to orchestrate when we have a very defined way of defining libraries. We have these build rules and things, and it's just like everything gels together in terms of the whole code base very well. So I can imagine it's going to be, as progress continues, use, it's going to be a very discreet moment where suddenly lots of libraries can be very quickly iterated on within our code base. [9:38](https://realalpha.app/v/atMRWzgHEGg/?t=578) Yeah. You've got your AI engineers proposing experiments to you for what's good to try. It seems that these models work super well in easily verifiable domains and coding and math have obviously been among them. How do you think about how, for some of these less easily verifiable domains, these models may end up scaling and being useful? I mean, they're getting better at that stuff too, but yeah, it's definitely harder. I guess in those domains, we're going to need either better ways of verifying or more human feedback loops. Yeah, I think what's good is that we're seeing is like, even with the Gemini model series, they're able to follow much more abstract instructions. So being able to try and provide a reward signal over a qualitative piece of work, which maybe if a human were going to try and give a feedback signal, there would be a quite a broad set of like rubrics or grading criteria. [10:42](https://realalpha.app/v/atMRWzgHEGg/?t=642) or maybe there is even more like a even kind of simplistic sense of like what is good style what is like interesting and and so i i think part of the problem is really training models to like [10:55](https://realalpha.app/v/atMRWzgHEGg/?t=655) **Jack Rae:** take on these like very broad criteria for what uh how to how to like take in a very broad set of criteria and then apply a reward signal once we have the reward signal we can train with reinforcement learning against it and uh and i think yeah we're already seeing that kind of like [11:11](https://realalpha.app/v/atMRWzgHEGg/?t=671) makes makes sense it's not like an abstract thing anymore it seemed like a very abstract thing maybe a year ago or two years ago i'm curious a year or two ago like did you expect that to work or do you think this like whole path of of research was really more toward these like more easily verifiable domains i feel like you expect it to work one day and then it feels like there's a very there's a very complicated stack of things that we feel like we need to solve to get there and then usually the case it's like oh it turns out there was a much simpler path that's how i feel about it anyway yeah usually there are good surprises you know more good surprises than bad surprises you've released these models out in the wild um what are some of your favorite kind of like ways you've seen people using them um and what would you like to see more people like trying and building with these things actually there's just like an update today happening in the gemini app where we're putting in a much stronger model and it's being integrated with basically the full suite of like Gemini app tools so it's like all of the apps that Gemini app supports uh from things like you know maps uh also like search integration and now has [12:17](https://realalpha.app/v/atMRWzgHEGg/?t=737) like very long context all these things it's kind of like a fully featured Gemini thinking release in the app and it is actually you know it's a very enjoyable experience that's what I think so far people even for their day-to-day stuff people uh it wasn't clear to me whether they would like to have to pay a bit more latency to like get the model to think about stuff before responding, it does seem to be the case. But if people are going to like pull their phone out of their pocket and then type something in, that actually what we thought was a very long time, maybe a couple of seconds, is like a very small price to pay for something that they feel like is a better quality answer. [12:50](https://realalpha.app/v/atMRWzgHEGg/?t=770) And also maybe they can like sometimes look at the thoughts and inspect them. Like I was kind of showing this to my mom like a couple of days ago. Like moms are the ultimate test of whether like something has broken the barrier from like the Twitter sphere to like the real world. Yeah, like the mum vibe check is a big deal. And yeah, I guess she asked a lot of what I would consider very generic questions to ask a model, like what is the meaning of life? Oh, meaning of life. She went for what is the meaning of life and she really like sat and read it for a long time. And then she read the thoughts as well and then she kind of like contemplated on it and she felt like, I don't know, she seemed to kind of very much appreciate the presence of like thoughts as to like how to even go around such like an open-ended question. [13:36](https://realalpha.app/v/atMRWzgHEGg/?t=816) So more folks building philosophical conversations with these models. Well, I mean, one thing I think is super impressive about them is the multimodal capabilities, but it still feels like those are very underexplored from an application perspective. Not sure totally why that is, but... Yeah, I think right now, in some ways, we are quite modest about the multimodal capabilities of Gemini. I feel like the model has always been incredibly strong at image input. Image input plus thinking is actually remarkably good, I would say. I see a lot of people kind of red teaming the model on things like OnX and trying difficult or challenging images, visual reasoning problems. [14:15](https://realalpha.app/v/atMRWzgHEGg/?t=855) I think that is working pretty well. There is also, at some point, some of those are kind of toy evals, but pulling in multimodal with then agentic tasks is super interesting as well. So, like, we launched Mariner last December, which was, like, an agent which, like, uses a browser and things. That has a lot of multimodal, like, aspects all built in. It was super important that they could get the model to be incredibly strong at, like, not only kind of scanning a screen, but, like, really understanding it and knowing how to act on many, many different types of websites. So, like, pairing that kind of capability, agentic, quite open-ended, maybe you really need visual understanding of potentially messy scenes with thinking is something that I think I'm feeling very excited about. [15:02](https://realalpha.app/v/atMRWzgHEGg/?t=902) Yeah, that's one. I mean, it made sense to start with text. You know, a picture's worth a thousand words, but it's a million pixels, so the text is still a thousand times more information-dense just to do math on cliches. But and then there's also like a lot more just in the main training data for text. You know, we have so many examples of text, which actually kind of represents, you know, the way humans receive and produce information, you know, through language. But, you know, we have a lot fewer examples where for something like, say, image generation, because you don't have examples of people generating images. [15:50](https://realalpha.app/v/atMRWzgHEGg/?t=950) So things are a little more challenging, but we, you know, it's, as Jack said, you know, great stuff, great stuff happening. Yeah. When you mentioned Project Mariner, I'm curious, like, you know, I think you both talked before about, you know, to get really, you know, agents more widely used, you kind of need to solve both complexity of reasoning and then also reliability. How would you kind of characterize where we are today in terms of applying, you know, Gemini models toward these problems? and what is the actual path to getting better at a bunch of these things? I mean, there are a lot of answers. [16:20](https://realalpha.app/v/atMRWzgHEGg/?t=980) I mean, one is just make the model smarter. That will always help. And, you know, most likely, you need very general solutions for... For, you know, for these control problems, you know, just like you need general solutions for the intelligence problems because, you know, people are going to use these things in so many unforeseen ways. You know, like you cannot, you can't anticipate it. The users are smarter than the developers in figuring out what the use cases are. You know, no one envisioned like when they invented the internet, like what the internet was going to be for. No one envisioned when they invented computers, what computers are going to be for. [17:20](https://realalpha.app/v/atMRWzgHEGg/?t=1040) And then, you know, AI is, you know, is getting so, so general these days that it's even more true. We're just building a product with billions and billions of use cases that are unanticipated. So we will need to build the general solutions for all of it. How far away does it feel like we are from that kind of like next level of complexity and reliability? I don't want to give a precise like, is it six months? Is it 18 months? Is it 24 months? But I think a lot of really the time actually is not so much dependent on the, in my opinion, is like not really about like the core algorithmic AI development. [18:10](https://realalpha.app/v/atMRWzgHEGg/?t=1090) There's part of it is just like there's a lot of even like engineering challenges to really changing the whole way you train to being in more complex agentic environments that that has like some like almost constant time but non-trivial cost to like switch how we do research. So I feel like agentic research, it seems like a lot of the upfront challenges like it's no longer going to be very simple prompts and responses. we're now going to act in an environment so how you define those environments that kind of angle though has at the very least been something that DeepMind has been brought in on even since I joined in 2014 it's always been about there's part of building AGI which is like figuring out a really good agent and there's part which is figuring out a really good and general environment [19:01](https://realalpha.app/v/atMRWzgHEGg/?t=1141) **Jack Rae:** I don't think anyone has solved the perfect environment yet and you know [19:06](https://realalpha.app/v/atMRWzgHEGg/?t=1146) there's some obvious ones there's a notion of like using like a web UI and being able to automate many web tasks. There's a notion of having a code base and being able to work within that code base and do many useful things. But I think picking out those, if you can pick out a really good environment, then we can accelerate a lot of agentic research in that environment and build really good algorithms. And I feel like that's as big a part of the challenge as any given breakthrough in attention or in long context or reinforcement learning. How high do you guys think the ceiling is for continuing on this like test time compute vector? [19:43](https://realalpha.app/v/atMRWzgHEGg/?t=1183) I mean, obviously you've, you know, I think very publicly like Ilya's come out and said there's this entirely, you know, new direction that's needed to really advance AI to the next level. Do you agree with that? Yes, I pretty much agree because LLMs are just too cheap. Like, you know, like if you, what, like operations cost, you know, under 10 to the negative $18 these days. So, like, you know, if you can infer relatively efficiently, even on a very large model, you're, you know, you're getting like over a million tokens per dollar. You know, if, you know, I guess you can just check the prices on Gemini or anybody else. [20:28](https://realalpha.app/v/atMRWzgHEGg/?t=1228) Yeah, so you're getting millions of tokens per dollar. That's orders of magnitude below the cost of most other things you can think of. [20:38](https://realalpha.app/v/atMRWzgHEGg/?t=1238) **Noam Shazeer:** If you think of like a really cheap pastime, like go like buy a paperback book and read it or something, you're paying like 10,000 tokens per dollar. [20:46](https://realalpha.app/v/atMRWzgHEGg/?t=1246) So we're like a couple of orders of magnitude cheaper than, you know, than like reading a book and, you know, probably, you know, four orders of magnitude cheaper than, you know, paying anybody to do anything. and, you know, whatever, six-to-eight orders, magnitude cheaper than paying a software engineer. So there's a huge, huge margin of difference there to apply more compute and make the thing smarter and, you know, if the value is there, which I think it is. Like, you know, would you pay, you know, five cents an hour for the bad engineer or ten cents an hour for the good engineer? You know, like it's there's so there's a huge amount of sort of unexploited flops in there for to use if we can find ways to use them. [21:54](https://realalpha.app/v/atMRWzgHEGg/?t=1314) And one way to use them, straightforwardly, is, OK, just train a bigger, better model. We're already doing that. But still, model training costs tend to go up quadratically with the size of the models. So then you still end up with a relatively cheap inference, if you do it right. So then, of course, what everyone is doing now is just apply more compute at inference time through this chain of thought thinking or any other brilliant algorithms we can come up with. And so I think we're just going to start seeing a scaling curve there as well as we're seeing in a lot of places. Right. But does that scale us all the way to the AGI future people envision? [22:52](https://realalpha.app/v/atMRWzgHEGg/?t=1372) or is there some, you know, some kind of completely adjacent thing that's required to, you know, I guess where does an ascent tip? Yeah, I guess that's where we'll see whether, you know, whether it's the humans that invent the next breakthrough or the AI. But I'm not, I've given up on like organizing my garage and stuff like that because I'll just wait for the robots. Yeah, you think they're coming? I guess you guys had a big robotics release yesterday. Yeah, that was awesome. Yeah. Is it ready to clean your garage, though? Not that I know of. Yeah, I think the question of how much more is there to give in this test time compute paradigm? [23:31](https://realalpha.app/v/atMRWzgHEGg/?t=1411) Is it all the way to AGI? I don't think it's all the way to AGI. I think we already kind of established there's other components, being able to act in a complex environment. Acting is very important. Research into acting agents, that's a definite investment. And there's many other aspects. But are we seeing kind of test time compute asymptoting? I think like the kind of Ramanujan example is always in my mind here where it's like I don't want test time compute to just for any given problem like think longer and then eventually arrive at solution. But we also want it to be able to think like very deeply and actually like create maybe useful knowledge that it's going to like actually incorporate to then solve further tasks in its thoughts and thus improve dramatically improve data efficiency. [24:17](https://realalpha.app/v/atMRWzgHEGg/?t=1457) if you can just have like one math textbook and you spend most of your time really just thinking and playing around with the ideas and then you can become a world class mathematician that would be the kind of thing that I feel like you know should be and we should strive to achieve with like a very deep thinking model are we there yet no like do we have a path there. I think there is. There are many directions of gradient towards such a model. We've been already seeing, and this is one thing I think people don't really talk about too much with test time compute, but we're already seeing amazing improvement in data efficiency by training the model to think deeply with reinforcement learning when it's solving the task. So even if we're going to have a bunch of RL data, we're not going to add any more, even just like going, the test time compute paradigm is allowing us to learn a lot more from that data. And I think we could probably push that much more so. There's kind of many particular research angles we're interested in and how this model could think not just in one particular, you know, spitting out a couple of thousands of tokens in order to solve the task, but maybe like think much more deeply, much more kind of like a researcher might think about a hard [25:27](https://realalpha.app/v/atMRWzgHEGg/?t=1527) and open-ended problem. I think so much of what you guys have touched on is like these models increasingly acting like researchers. What are like the early signs of that that you would look for? I think like, you know, I'll give one example, which is like math. So I think right now math is being treated as a, it's a bit strange and math is often used as benchmarks and they kind of like exams and maybe even like math competitions. There's going to be like a very important pivot from the math benchmarks and to being really starting to become about like actually generating useful math, starting to solve actually important problems that we really care about um i think there are ways like i i think it was very cool this like frontier math uh eval that was created which is trying to provide like a kind of a gradient towards that so maybe like the harder category are basically almost like unpublished math findings the easier category is supposed to just be like kind of harder or trickier you know i don't know whether that particular eval is the perfect way but having some kind of ramp of evals that like bridges from where we're at now to like actually useful scientific contributions, this is something I'm like especially excited in [26:34](https://realalpha.app/v/atMRWzgHEGg/?t=1594) as, you know, bringing in professors and researchers and saying, okay, use this tool in a non-ironic way to actually accelerate research in this area. What would you do? What's missing? What do we need to advance? And I think, yeah, I think like this kind of notion of like incrementally harder benchmarks, it might sound like, oh, that's like what AI researchers always say, but I think that's going to basically be my metric of progress. I mean, math is a great example here because, I mean, this is a field that you don't actually need more data. Like, you know, like people invented all this math without this external input or a world up to it. [27:11](https://realalpha.app/v/atMRWzgHEGg/?t=1631) Yeah, a lot less data. Oftentimes in like a room, just thinking of your things. Yeah, you just go into a room. And then there are examples where, okay, there's some data, but, you know, okay, Isaac Newton like takes a bunch of astronomical observations, like goes into quarantine in his house and then like invents physics or something. So, okay, that's an example where there's data but nobody actually knows physics and then you generate physics and then math is even crazier because you start with roughly no data and invent something useful. So that kind of could provide a counterproof to the assertion that, hey, this is just learning to mimic people. [28:02](https://realalpha.app/v/atMRWzgHEGg/?t=1682) That, okay, the most we can do with AI is to relearn what everybody knows and what we've... The learning to mimic people critique, I feel, has been around for a while. I feel like Jan Koon said a lot about this. Do you feel like that's entirely disproven at this point? basically the critique that like uh you know novel uh discovery and thinking is kind of impossible in these current model architectures well there's definitely one class of scientific discovery that i think almost no one could argue against which is like actually a lot of science is like if you knew about these two disjoint pieces of information and thought about the intersection For a while, you would realize a new property, like in kind of material science, it might be like, oh, it turns out just like associating things way better means actually, you know, way more about like what may, you know, what new kind of material may be photovoltaic, but also may, you know, etc. [28:54](https://realalpha.app/v/atMRWzgHEGg/?t=1734) So there's like interpolation already would actually completely accelerate science. And I think I'm guessing then it's always going to be this kind of whack-a-mole of like what is considered actually interpolating known ideas versus like creating a completely novel idea. And for that one, I'm going to go to NAMM. Yeah, I, you know, I don't know, but I guess I'd just throw it back at Jan LeCun to prove that he generated a completely novel idea. We'll see. I don't actually care about arguing this stuff. Let's, you know, let's just, you know, build AI, you know, greatly increase the level of technology in the world, help people that seems good enough for me. [29:43](https://realalpha.app/v/atMRWzgHEGg/?t=1783) Even if we go back to the math example, like the thing I'd love to see is like, you know, maybe the state of mathematics right now is kind of like the state of kind of geographic exploration in the 15th century or something. So it's like, there's some known things, there's some like, fuzzy area of like, we don't really know what's like beyond these boundaries. And we have a bit of a guess. And then you send a small number of people to go off in a boat, it's very expensive to try and like, push the boundary of like, what what is known and what is not known. And it feels like some pretty funny looking maps. [30:13](https://realalpha.app/v/atMRWzgHEGg/?t=1813) Yeah, they come back with like a little bit of extra territory, like explored, and then they tell people. And that's a little bit like what's happening right now in mathematics. You have a very small number of like elite math professors that are able to really ask the right questions and then actually like prove useful qualities. And it kind of grows and it's grown that way. And that's how it's been going so far. right if you if we can train a model that can actually like essentially ask like it can pose the right questions of what you could say is like the space of all uh useful mathematics is kind of an infinite space you don't want this to go and like off like a fractal on on kind of uninteresting questions but if there was some notion of like all the set of like interesting mathematical questions if it can like keep posing new ones and then if it's very very strong and it can solve those, then at some point, you know, maybe like right now we have a pretty strong kind of math model. And at some point, maybe it will be a professor level. Maybe one day it'll be Terry Tao level. And then you have like a million of them. And then now it's like, now I think then you could hope to maybe complete the map. And then what would like completing the map look like? That [31:18](https://realalpha.app/v/atMRWzgHEGg/?t=1878) could be one of the greatest contributions to kind of science. You would now physics, chemistry, you'd now be able to have a very deep understanding of any useful mathematics. I think that would be a very, very exciting thing. [31:33](https://realalpha.app/v/atMRWzgHEGg/?t=1893) **Jack Rae:** Whether it's possible, I'm not sure, but I would agree maybe the key crux is can I ask actually novel questions? [31:42](https://realalpha.app/v/atMRWzgHEGg/?t=1902) The question posing thing seems to be the hardest part, in my opinion. The solving, I feel very confident we will get there. But maybe mathematics is infinite. Yeah, I mean, it's definitely infinite. Yeah, but we can do so much better. Yeah, yeah. I guess let me just talk about the culture of AI research a little bit. Like, you know, Noam, you were obviously part of the, you know, a leading part of the original Transformer paper. Both of you have been a part of so many breakthroughs. You know, people like to write these thought pieces about, you know, the culture that drove these innovations. Curious, like, what you think the main takeaways you have after, you know, having done this research, you know, some periods of great success, some periods maybe of more frustration. [32:17](https://realalpha.app/v/atMRWzgHEGg/?t=1937) like what lessons do you take away from what actually you know works from a cultural and team structure perspective to drive this stuff forward maybe maybe uh sort of ai research is kind of where uh chemistry was in the 15th century same alchemy it's alchemy like we don't know why it works like but it's it's highly experimental uh You know, in terms of, you know, you get some idea, but really the proof is always in try it out. And then, you know, and then you have various observations. You come to hypotheses about why, you know, okay, this thing works. You know, why? What's the key thing? And, you know, so sometimes you're right, sometimes you're wrong. [33:05](https://realalpha.app/v/atMRWzgHEGg/?t=1985) it usually takes takes more more experimentation to find out or you can just like you know just claim that okay it works because of my magic XYZ and you have your assistant swallowing frogs or something and or the or the equivalent you know I find you know naturally researchers just, you know, love to share and get excited about, you know, about what they're doing. And, you know, it is always important to try to, you know, credit people, you know, liberally because, you know, it's, you know, it's often very complicated to, you know, okay, what idea led to what idea, what was the key insight. And I think at some point, we'll just kind of have to give up on credit assignment temporarily and take a super intelligence-based approach of, we'll wait for the super intelligence to sort it all out once we get it. [34:17](https://realalpha.app/v/atMRWzgHEGg/?t=2057) Super intelligence will write some great thought pieces on the culture that drove these transformations. But it's super fun working at Google with this group of brilliant people and collaborating with everybody. One thing I'm struck by the Transformer story, I think, is it involved, I think you heard randomly that some people were working on it. When you tell the story, it sounds like, wow, that could have easily not happened. I don't know. If you weren't walking down that hall one day, does it feel like inevitably someone six months later would have figured that out? How much just random happenstance is there that kind of leads us toward a different path from just people in this building colliding in different ways? [34:50](https://realalpha.app/v/atMRWzgHEGg/?t=2090) Oh, interesting. Yeah, we would have all been using LSTMs or something. I mean, I guess maybe it's like, okay, you ask the same question about like, okay, if somebody hadn't invented like an internal combustion engine, would we still be using steam engines at this point? I mean, I think someone would have come up with something like Transformer. From my vantage point, like, over in London, it did feel like you were circling around the, there was, like, neural GPU, there was, like, which was also, like, okay, the key thing is we're not going to have an RNN anymore, we're going to paralyze, but it's not just going to be a conf net, we're going to have, like, a notion of, like, the, I don't know, like, depth is going to be some kind of function of your sequence length, that was an idea in neural GPU, it didn't quite work out. [35:41](https://realalpha.app/v/atMRWzgHEGg/?t=2141) But it was, like, the key idea of, like, get rid of RNNs felt like that vibe was there. Yeah, right, because there were all this work on confnets running around. Everyone wanted to kill the LSTM. And, yeah, tension had been floating around from the translation models. So it kind of needed to come together. I love that. also kind of on the culture side, I think one of the interesting things of how I understand Google works is basically there's like this bottoms-up compute allocation, right, of folks, you know, getting to do different projects and then convincing other people to kind of come and allocate compute for that. [36:23](https://realalpha.app/v/atMRWzgHEGg/?t=2183) And obviously, that's one model. There's other places that are like, we're going to go all in on like one thing and they're much more top-down on the compute side. How do you think about like the trade-offs between those two models? Yeah, I mean, we've been through both or I've been through both, I guess. Google brain When I was at Google previously was, you know, mostly bottom-up, you know, as you describe, and then DeepMind has been mostly top-down. Yeah, it's a bit more top-down. A bit more top-down and, you know, different philosophies. And, you know, there are pluses and minuses to both. I think, you know, top-down can be good for getting people to collaborate and getting, like, larger training runs working, but bottom-up I think is also great for collaboration because then you bring someone new on your project it doesn't mean you have fewer resources per person, you have more total resources so that's great [37:23](https://realalpha.app/v/atMRWzgHEGg/?t=2243) **Noam Shazeer:** and that there are so many abstraction-breaking ideas that there's no great way to categorize them [37:31](https://realalpha.app/v/atMRWzgHEGg/?t=2251) so if you're saying, okay, this is the compute for pre-training and this is the compute for post-training, well, okay, you've got something completely different that doesn't fall into those things nicely and then it falls between the cracks. So we're bringing back a good measure of bottoms up because I think that's super important. Yeah, it feels like an interesting balance to strike. Yeah, I guess I worked at OpenAI briefly, and I did like the way that concentrated bets were made and that obviously paid off well for them in certain areas, and that was nice. I also liked the way concentrated bets were made at DeepMind, especially kind of like, yeah, I do think there was a bit of a vision sometimes for like, say AlphaGo, I was asked to join the AlphaGo team right at the beginning because they needed an engineer and I was a research engineer and I just didn't get it. [38:29](https://realalpha.app/v/atMRWzgHEGg/?t=2309) I was like, why would anyone care about a board game? I don't get it. So you do really need like good research leads that have vision to drive these things. You can't expect everyone from a like, that doesn't have always the best like vantage point to like know what's, where the impact is going to be. Right now within like, for example, thinking as like the area of research that we kind of work within, it's like incredibly important. We have a reasonable non-trivial investment in like just bottom up research where we're really not dictating anything. And then it's just a fun process of, it's kind of a fun meta research process of like how can we make these people maximally efficient? [39:04](https://realalpha.app/v/atMRWzgHEGg/?t=2344) How do you make your baselines lightweight? How do you make these signal bearing? How can people move as fast as possible? And that's very important. And then it's just very important that it can't all be that. It needs to be, and here is a mandate of top-down bets that we have to deliver on. But it's always humbling. I think we have a very good scope of what everything is happening and we get a sense of like, here are the areas that are going to be really important. And usually you get humbled by the bottom-up research of a thing that you weren't even thinking about ends up being way more impactful than you thought. [39:34](https://realalpha.app/v/atMRWzgHEGg/?t=2374) So just always keeping that running is super impactful. Yeah, I mean, reflecting back on the past decade, are there some of these inflection points or these decision points where it's maybe hard decisions, 51, 49, that ended up being kind of super impactful? I mean, I think for both of us, going in on large language models was good. Yeah, I'd say that was probably a good call. and it kind of seems obvious now but it kind of was at least I mean I found it was like in a state of being not obvious to most people but obvious I felt like to a small number of collaborators that like this was definitely gonna be a big thing and kind of had to go against the grain for a while. Yeah that's true there was definitely a time people were not excited about language models. Hard to remember now it feels like decades ago. I mean it always seemed like the best problem on earth to me but you know uh that's yeah it's kind of like the deep learning people at the time maybe thought machine translation was still a bit cooler or computer vision yeah vision was exciting for a while i don't know why was everyone into vision No, I guess, like, the ImageNet thing. [40:39](https://realalpha.app/v/atMRWzgHEGg/?t=2439) Yeah, or there's, like, a picture of a cat or something like that. Oh, that was, like, good one, the other cat. Did you work on the cat thing? No, no, no, never actually did much in Vision. I've run M&S. And now you've come full circle on these Gemini 2.0 models. I see all these demos that are, like, Vision-based, and you're kind of doing cooler things in Vision now than certainly identifying cats. I mean, it felt like, I think, even almost every early LLM person, world model is on their mind, really. I actually don't feel like the early LLM researchers were even really from a language-oriented background. [41:15](https://realalpha.app/v/atMRWzgHEGg/?t=2475) They weren't really like linguists. It was more, I don't even feel like understanding language was really part of the motivation for that early group. It was like train unsupervised learning at scale, do language first because it's the most like knowledge compressed, but then gobble everything up into like a big generative model and like understand everything. And it's very, very cool to see that just, like, continually proving out. Yesterday, we just, like, launched native image generation. It's amazing. You know, I think a lot of image generation right now is just, like, focused purely on getting, like, absolutely maximally aesthetic images. [41:50](https://realalpha.app/v/atMRWzgHEGg/?t=2510) But also having native image generation allows you to, like, really do a lot more with images, understanding, editing. You can have, like, interleave sequence of images and text. And, yeah, once again, it's just train a generative model on lots of data to arrive at that. You guys are obviously both big believers in these models becoming more and more general purpose. I guess a question some folks are asking is, you think about domains. You talked about healthcare earlier, Nome. The model that ultimately is going to be our AI doctor, is that just a continuation of what we're doing in some giant model? [42:24](https://realalpha.app/v/atMRWzgHEGg/?t=2544) Is there, like, a healthcare-specific version of the model that ends up being released that is, like, you know, only has some set of data, you know, inputted into it? Or they're just a bunch of guardrails? Or, like, paint that picture for me of what you ultimately think, like, our, you know, AI doctor or AI, you know, biology researcher. Yeah, I don't really think you, you know, would need very, you know, specific, you know, test-specific models for something that high value. because, you know, you probably pay like $1 a token for talking to your doctor. So it's like the LLM is like way, way, way cheaper at this point. [43:06](https://realalpha.app/v/atMRWzgHEGg/?t=2586) So the only reason for task-specific models is price. And so if there's things that you wouldn't pay a dollar a token, yeah yeah yeah like you need targeted yeah something to analyze you know vast quantities of data for marginal value then maybe you want something yeah you can test specific there's always this notion of like there'll be tons of like negative transfer out there so like compartment things i don't really feel like that has ended up being the case if it could be measured then And that's a good reason to compartmentals, like models. If there's no negative transfer, there's positive transfer, then just have one big model. [43:43](https://realalpha.app/v/atMRWzgHEGg/?t=2623) That's like, I think, my personal philosophy. This is not like a thing which people have like uniform agreement over, though. It's like a continuous active area of research. Like how much do you want to like specialize and spin off these expert models? But yeah, from the way I see it, it's just very simple. [44:00](https://realalpha.app/v/atMRWzgHEGg/?t=2640) **Jack Rae:** If there's positive transfer, put it in the same model, as long as it doesn't then become too expensive to serve. [44:05](https://realalpha.app/v/atMRWzgHEGg/?t=2645) You know, obviously you guys have been at the cutting edge for a while. What's one thing you've like changed your mind on in the past year? I feel like timelines have shifted forward. And I don't mean that like in a vague sense. Like I think the rate of progress is much faster right now. I felt like a year ago, there was obviously the field is advancing. But whenever you have a new paradigm shift, it creates this sudden acceleration. And also, actually, no, this is a pretty. Good one. One thing I've changed my mind on, like, my mental model of, like, the propagation of information and how, like, how people, like, adopt a scientific advance has completely changed. So when the Transformer came out, I think, so I was over in London at DeepMind. People thought it was a cool paper, were a bit suspicious. I eventually implemented it in our, like, code base over the, like, holiday break, actually. But it was, like, three months after the paper had come out. [45:06](https://realalpha.app/v/atMRWzgHEGg/?t=2706) I eventually implemented it and tried it for like language modeling but it was like not really getting picked up I eventually then collaborated with someone that wanted to use it for reinforcement learning but I'm going to say really it was like from paper coming out maybe it was like six to nine months before it was like you saw just transformers dotted around all areas of deep mind and that's within out like we're all within alphabet and it's like much easier to propagate information um I would say the the the speed at which the field has like picked up this test time compute paradigm you have like many labs have already trained and released models that like looking very good exploring the space that was like very surprising to me the fact that these things can like if you make an announcement say and like this is important the fact and it's just like a blog post or something and then you'll have people that are able to like uh make breakthroughs in that space and release models in order of months that is like that was a wake-up call for me there's a lot more compute and a lot more smart people working in the ai i often think of things in a bit of a rose-tinted glasses way and think about like 2016 or something and i think well [46:14](https://realalpha.app/v/atMRWzgHEGg/?t=2774) we're very smart then we're very creative people are very very smart and creative now and they have way more compute and there's way more of them and so if anything is going to like be very impactful and then it can just like spread and like this idea can like spread all across uh all across the world and people act on it which is kind of crazy yeah it is it is kind of crazy and just the amount of compute out there that yeah that like now uh you know whatever a kid in the garage has more compute than was necessary to invent the transformer so like uh you know um yeah i mean people always worry about oh just how much can compute do they have but you know it is it is definitely possible to make breakthroughs, you know, with way less compute than you would imagine. [47:05](https://realalpha.app/v/atMRWzgHEGg/?t=2825) Yeah. Anything you've changed your mind on in the past year? I mean, I've been continuously impressed with the success of RL. I'd never really worked with it much before. It's like, oh, that's actually pretty good. Well, I mean, you kind of alluded to this, Jack. Like, obviously, I think in the reaction to DeepSeq and all these kind of models that kind of fast-followed in the test-time-compete space, going forward, do you expect the open-source models to be able to keep up with each subsequent generation of these models? It obviously seemed to happen faster than a lot of people would have expected. [47:42](https://realalpha.app/v/atMRWzgHEGg/?t=2862) Yeah, that's actually something I'm changing my mind on. I do feel like the open-source, like the ability for open source models to stay very close and competitive with the frontier is persisting. I actually thought we were getting maybe a false sense of assurance that it's happening because maybe it felt like it was converging, but then these things can pull away again. But actually, that seems to have been very impressive. I'm really, really impressed at the performance of Gemma 3 that just got released yesterday. It's amazing. It's completely incredible. The team did a really good job. And other open source models. [48:22](https://realalpha.app/v/atMRWzgHEGg/?t=2902) Yeah, so, you know, DeepSeat V3 was a very good model when they released it. Yeah, so, it seems like people are very, like, passionate in the open source space. And they're very creative and smart. And they have compute. So, I don't really see why. why they wouldn't be able to continually innovate. Yeah, what do you think, Naan? Yeah, I mean, it seems like the time gap between closed source and open source is, you know, has been shrinking. You know, I think that the technology will continue to accelerate. So, I mean, it could be that the quality gap will be large and the time gap will be very small. [49:05](https://realalpha.app/v/atMRWzgHEGg/?t=2945) But, yeah, we'll have to see how it plays out. It's super exciting to see all of these companies getting great results. Switching to some of the broader implications on all the AI progress we've been talking about for society. I'm curious. Obviously, it seems like both of you in the last year have been impressed with the power of RL. You've been surprised by the pace at which we've scaled a lot of this test-time compute. um have you like changed anything in your own lives like based on kind of this you know probably uh more clarity or or or belief that you know a lot of this ai driven future is coming like you both have kids i know like any anything that that you've adapted uh it sounds like no i know you don't clean your garage but i guess that was a prior to this year as well you thought about not cleaning yeah yeah i don't worry too much about global warming you know we'll have ai to take care of the carbon stuff soon enough anyway you know yeah but i guess you know any anything that you've uh that you've thought about differently in your own lives or you know uh in how you think about the life your kids will have ai and education is like uh i didn't think people are really talking about it enough yet my son like under supervision but he likes to talk to gemini it's [50:18](https://realalpha.app/v/atMRWzgHEGg/?t=3018) actually insane like how powerful it is especially if he can like he goes out to the garden he like takes pictures of plants takes pictures of lizards and you and like he now has this like very accurate personalized encyclopedia which can give him information and do they adapt to that like i wasn't sure how that would work they do my four-year-old son walks around like talking very detailed about like the plants he'll use the latin name he's like they absorb so much stuff they're sponge i feel like i'm seeing kind of like a type of education that i don't really think has ever existed for like humanity happening like ai and education is going to be incredible uh he went to school and he was like um he was like oh yeah i caught a lizard to his teacher and his teacher was like oh that's cool um and it's like oh that looks like a big lizard he's like no it's not a big lizard it's a western fence lizard very particular about his type varieties of lizards And he's like, and I also saw a blue-tailed skink. [51:13](https://realalpha.app/v/atMRWzgHEGg/?t=3073) And she's like, what's that? He's like, it's an amphibious lizard that, you know. So he starts like kind of reeling these things out. And, you know, that's just like, it's just obvious when you see it, but children are very curious. They're like sponges for information. And if you can like combine that productively with AI, I think that's going to be really incredible. I do feel like the next generation will just seem like smarter people. That's what I'm feeling hopeful about. No, anything you've changed? Yeah, it is extremely hard to predict what the future will be like. We will all do our best to make sure that AI will be safe and beneficial. [52:03](https://realalpha.app/v/atMRWzgHEGg/?t=3123) But, you know, it does, you know, it does make you think that, hey, you know, what I do now really, you know, really, really matters, you know, like, okay, we don't know if, you know, human labor will be, you know, like, materially necessary in the future. but uh that that just means a you know like uh you know it makes uh makes more difference uh well if you want if you want to make uh if you want to do something uh that matters materially go uh go do it now and then other than that like uh you know just uh try to be a good person you know whatever you uh find uh find uh spiritually uh meaningful uh you know go do it because that's that maybe that's uh uh you know the purpose of humanity uh uh in the future uh you know may not not be about providing for physical needs. [53:08](https://realalpha.app/v/atMRWzgHEGg/?t=3188) So we've got to figure out where we find meaning in the future. But we'll have plenty of time for, you know, the relationship or something else. Yeah, well, especially if your mom's already having the deep philosophical conversations with the models. Maybe we'll be able to reason our way to do that. But I am struck. I mean, I feel like some other people have come on this podcast, you know, Bob McGrew, the chief research officer of the AI was like, look, you know, the humanity is always going to have a role in asking the questions. The models will go off and do things. But I think to our conversation earlier, it's like, I think the big question is, will, you know, people always be the best people to ask questions? Or, you know, will the models actually ask better questions over time? [53:47](https://realalpha.app/v/atMRWzgHEGg/?t=3227) And obviously, it has a ton of implications. I feel like every generation thinks they're living through the most important moment in history. But it does, you know, I guess, you know, you're biased when you're going through it. But it does feel like we are certainly in that. Yeah. And I think sometimes like a technological advancement kind of scares people and it is people have right to feel trepidation at this stage as well. But there is like, I mean, this isn't such a good example, but like even like the introduction of the, like the television, people are like, oh, is this going to make us all just like lose our attention span? Are we going to like completely lose the will to like go out and walk outside and like have friends and things? [54:22](https://realalpha.app/v/atMRWzgHEGg/?t=3262) that was like a thing people freaked out about i think uh obviously now we know that was like unnecessary there was like it was a small piece of technology that was entertaining maybe even has provided like a net positive to society i think it like kind of went okay right now it's like there's like it's it's like that but kind of on steroids there's like very strong signs of how this helps us there's like very good reason to be concerned about how this could not help us so it's kind of like you you have like very demonstrable reasons that have already proven out of how it's helping us you have very like concrete arguments for how this could not go well and i think that makes it a very interesting time too in some ways you know we have less uh kind of um meaning than than in the past in the sense that like uh you know in the distance past everyone was like at the break of starvation and you're like okay i have meaning in my work because i have to go work hard today and get some money so my family doesn't starve tomorrow and like today we're living in America. Nobody's family is starving tomorrow if you don't work hard. So that, you know, okay, that's less meaning than we used to have. And we've found other, you know, [55:34](https://realalpha.app/v/atMRWzgHEGg/?t=3334) other sources of meaning. So, you know, future, more of that in the future as hopefully, you know, AI improves our physical situation. How worried are you both about AGI risks? I would say moderately, yeah. It is hard. It's often difficult to find examples of creating something which becomes far more intelligent than its creator, but then still acts in predictable and useful ways for its creator. And I think that that, like, class of argument is concerning. There's also just, like, more practical, like, kind of AI and society implications that we've kind of touched upon. But, like, making sure that AI is, like, constructive to the economy and the, like, and the, and people can kind of offload their lifestyle. [56:30](https://realalpha.app/v/atMRWzgHEGg/?t=3390) And we don't have, like, sharp changes in, like, the employment landscape and things. That's super important. Both of those are often on my mind. And then there's just very pragmatic things when we're always putting more capable models out. We're very excited as technologists to develop and ship things. But we also, I think we have a pretty good balance of like, of then internally having another group that's going to be thinking about this much more holistically. And like, how can we make this launch safe? What are some unintended consequences? Which I think has been pretty good and is like super important. [57:06](https://realalpha.app/v/atMRWzgHEGg/?t=3426) So I'm glad that happens. Yeah, I mean, we, hey, I agree. We need to, yeah, I mean, I'm not afraid, but we definitely need to be working on all the safety aspects of all of that. And there are examples of, you know, we create something that becomes smarter and more powerful in that, you know, we have kids. and they're smarter than us and or they're already detailed encyclopedias of amphibians and then they become teenagers and then you solve the alignment problem yeah yeah so I mean hopefully if we respect our parents and you know treat them well the AI will learn from us you have to have a lot of tokens on the internet of people being really respectful toward their parents exactly yeah we have to stop pushing the robots over respect for creators yeah exactly that that feels dangerous um you know one thing i did want to ask you about noam is obviously you know uh in your uh previous in between google stints you'd spent a lot of time kind of building out character and yeah a lot about kind of that product space of what like you know uh ai companions and the ability for folks to to chat with all sorts of different kinds of people what do you you know what do you feel about where that space is today what kind of [58:31](https://realalpha.app/v/atMRWzgHEGg/?t=3511) like problems, do you feel like we still need to solve there? Yeah, I mean, it's interesting because the main reason I left Google to start Character was I thought that the biggest thing that the LLM industry could use was an application where anybody can go and interact with LLMs and use them and discover use cases that were that were good for them. And, you know, because this was before, you know, before ChatGPT launched, before Gemini launched. So, I mean, mission accomplished. Everyone's out there talking to OLMs now. But, which is kind of different from how things are now. I guess the other thing was, you know, going into character, we were not really focused on, hey, this is going to be uh an entertainment product or something or something else we kind of just went in with an open mind and we're like okay um you know we're going to just put this out there in a general way we're going to help um help people conceptualize this as you know okay this thing can um you know take on different personas meaning it's a you know it's a very very general technology see what you can make it do. [59:57](https://realalpha.app/v/atMRWzgHEGg/?t=3597) And definitely, we found a lot of people are using it for entertainment, I think partially because at that point [1:00:03](https://realalpha.app/v/atMRWzgHEGg/?t=3603) **Noam Shazeer:** in the technology, like, OK, nobody's figured out how to make this thing not hallucinate. So people are going to use it for applications where hallucination is actually a feature, such as entertainment. [1:00:20](https://realalpha.app/v/atMRWzgHEGg/?t=3620) then, you know, that's, you know, I think that's worked pretty well. I know a lot of people like using character. Wait, what were you asking? I guess, like, what do you think the future of that? Obviously, like, there's this early behavior of people interacting. Yeah, yeah, what is the future? What does it look like, like, five, ten years from now? That's a good question. I do not know. I mean, I do think, you know, people are going to, I think people will always want relationships with humans because it, you know, it's spiritually more meaningful. But, you know, I think people will, you know, like having AIs that are kind of more in human form for, you know, for, you know, for things they want. [1:01:09](https://realalpha.app/v/atMRWzgHEGg/?t=3669) I mean, like, imagine, you know, you just get elected president. You get your piece in the cabinet to, like, advise you, like, wherever, you know, wherever you go. I mean, that could be cool. Yeah, you get your own personal AI cabinet. that you get your whole... A good AI summarization of all the secrets that he needs now. Yeah, or maybe you're a CEO of your own AI company, so you get a lot more productivity. I guess that's maybe in a lot of those cases, it's less about the personality and more about the productivity. But I think to the degree that people like interacting with something that feels human, probably we'll see a lot of AI that feels more human in various ways. [1:02:02](https://realalpha.app/v/atMRWzgHEGg/?t=3722) And do you think progress in that space is just about the models getting better? Or is there a whole other set of human-computer interaction product questions that need to be solved? That's a good question. I mean, I think the models getting better is pretty big. But then, yeah, part of it is for whoever is running the application to decide, okay, what are we going to let people do? I think users are probably pretty good at, will be pretty good at specifying, okay, what do I want for an interface? And it'll be mostly about, okay, do we want to let them specify that? Totally. Well, look, both of you, you know, fascinating conversation. [1:02:56](https://realalpha.app/v/atMRWzgHEGg/?t=3776) We always like to end with just a quick fire to get your, you know, take on some overly broad questions that we cram into the end. Sounds good. Okay. So maybe to start, you know, what do you feel like is overhyped in the AI world today and what's underhyped? I personally feel like the Arc AGI eval is overhyped. Oh, that's very spicy. Yeah, I think actually maybe the progress has been quite slow because a lot of researchers just don't feel particularly inspired to do like kind of these very specific types of puzzles. puzzles we did a lot of that in like 2015 2016 and then we kind of felt like yeah you can if you if you know the puzzle domain you spend a lot of time like fixing all the like actual bottlenecks like maybe acting on these large grids is like a little bit finicky and stuff and and then and then you make a lot of progress but but then you don't necessarily continue on like building something that's really agi and useful so i feel like i personally had a transition from going to from like all these like synthetic tasks like uh to then going to just like just model natural language and i felt like that was a dragging in in that direction was like a much more agi thing in in the long run anything else comes to mind for you to overhyped or underhyped i don't [1:04:14](https://realalpha.app/v/atMRWzgHEGg/?t=3854) know i think i think agi is underhyped agi lms are just still still massively massively underhyped I think people are, you know, I don't know, people are still, like, thinking about it like it's only, like, you know, that it's going to be about some silly trillion-dollar products, you know. I heard you say in another pod that a trillion wasn't cool anymore, quadrillion was... exactly you gotta put the dr evil up um i obviously you know it'd be a gross misallocation of societal resources to take you away from building models to like build applications but i am curious like if you were to if we were to say today go build an application like what what do you think are the most interesting you've talked about education before but any other areas that come to mind that you think would be fun to to go build uh you know apps on top of these models well i i do think um i think it's actually very cool how many like apps that have been trying to break into this agentic space and people then expose it and say, oh, this is just a wrap around a known model, but there seems to be a lot of value in actually having the right app experience if you want to actually have a model do something useful for you, [1:05:23](https://realalpha.app/v/atMRWzgHEGg/?t=3923) act and do something useful for you. So I feel like, yeah, I feel like, okay, in the agentic space, I think code is very crowded now, but I do think there's a lot of other things that I would find it useful for a model to automate for me that goes beyond maybe like a chat experience but it's actually going to be going out and doing useful things. Yeah, I mean, I'd say code is underhyped. I think it's huge. I think it's huge because, you know, for one, humans aren't even that good at it. Things like code and math are not super designed for that. And then it's one of these things that will help self-accelerate. [1:06:04](https://realalpha.app/v/atMRWzgHEGg/?t=3964) You know, if it builds an automated software engineer, researcher, and then it'll build the next better AI. So, yeah, the combination of, you know, of engineering and agentic, you know, something that can control surfaces broad enough to do the job of an engineer. I think if I were to focus on applications, that is what I would be focused on. How different will the infrastructure needs look for test-time compute models versus these massive pre-training? You mean in terms of hardware? Yeah, in terms of the hardware requirements, distributed data centers, all that. Yeah. I mean, it's a pretty rosy story, I would say, right? [1:06:47](https://realalpha.app/v/atMRWzgHEGg/?t=4007) **Jack Rae:** So if it turns out that building AGI becomes mostly an inference problem, an inference problem that can be much more distributed than maybe synchronous large batch training [1:06:57](https://realalpha.app/v/atMRWzgHEGg/?t=4017) that happens in pre-training. I think that means that we can be much more flexible with our compute. But yeah, it's going to mean we maybe don't mind the model training across data centers as much. Maybe it can be spreading out actors that are going to go off and get experience and send that experience back from many, many different data centers. they don't all need to be like have very strong fast interconnects um so that is also going to drive price down as well because then we might start to like really optimize towards uh such a setup which is intrinsically cheaper the cool thing is at google we we kind of have like this co-design link with the tpu team so we're always like feeding them like our like the profile of like how we're spending our compute which allows them to tweak the chip design and the data center design, you know, within a couple of years timeframe, which I think is really, really like kind of motivating. [1:08:00](https://realalpha.app/v/atMRWzgHEGg/?t=4080) The fact that you can be distributed as Jack said gets better. I mean, the thing that gets worse about inference than training is that you lose a lot of the parallelism in the transformer that, you know, that just naively using transformer, you end up memory bound, like looking at your attention keys and values for every token that you're generating. So there's a lot of great work to do in both attacking this from a model architecture. perspective and from a hardware perspective frankly so you know to get ourselves closer to the point where we're you know where we're taking like the massive computational power of of the chips we have and and making ourselves able to like fully apply that to to inference I want to leave last Where can they go learn more about you and what you guys are doing? [1:09:08](https://realalpha.app/v/atMRWzgHEGg/?t=4148) Yeah, well, you know, we have a new and updated Flash model that applies thinking that's considerably stronger than the last model we released in January. It's out on the Gemini app. I would definitely encourage people to try that out and give us feedback. We have been incorporating developer feedback, user feedback into each model series. So, like, I think that's a good, like, that would be one thing I would encourage people to do. what jack said yeah well thank you both so much seriously it's uh such a pleasure to be able to do all this with you real pleasure yeah You