# Inference has room to get much more expensive

Markdown copy of https://realalpha.app/s/inference-has-room-to-get-much-more-expensive/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Noam Shazeer](https://realalpha.app/builders/noam-shazeer/) · OpenAI; co-author of the Transformer · Designed multi-head attention in the Transformer, and worked on sparse Mixture of Experts, Mesh-TensorFlow and T5.
- Host: [Dwarkesh Patel](https://realalpha.app/people/dwarkesh-patel/) · Host, Dwarkesh Podcast · Co-wrote The Scaling Era, an oral history of AI; named to TIME100 AI.
- Show: Dwarkesh Podcast
- Episode: [Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI](https://realalpha.app/v/v0gjI__RyCY/), Feb 12, 2025
- Moment: 53:02 to 58:27 of the episode (5 min), playing on https://realalpha.app/s/inference-has-room-to-get-much-more-expensive/; the episode from there: https://realalpha.app/v/v0gjI__RyCY/?t=3182

## Quote

> “And so you're getting like **a million tokens to the dollar**, right?”
> — [Noam Shazeer](https://realalpha.app/builders/noam-shazeer/) at [53:40](https://realalpha.app/v/v0gjI__RyCY/?t=3220) · Dwarkesh Podcast

## Takeaways

1. A trillion operations per token can cost around 10⁻¹⁸ dollars.
2. Inference may be far cheaper than reading or hiring an expert.
3. Search can help models spend extra compute on harder problems.

## About this segment

Noam Shazeer argues that model answers can improve by spending more compute at inference, since it is already far cheaper than many human services. Jeff Dean explains why search is likely to be part of the solution.

From “Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI”, Feb 12, 2025. [Watch the full episode](https://realalpha.app/v/v0gjI__RyCY/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[53:06](https://realalpha.app/v/v0gjI__RyCY/?t=3186) One of the big areas of improvement, I think, in the near future is this inference time compute, like applying more compute at inference time. And I guess the way I'd like to describe it is that, you know, like even some giant language model, you know, even if you're doing, say, a trillion operations per token, which is, you know, more than most people are doing these days, you know, operations cost something like 10 to the negative $18.

[53:43](https://realalpha.app/v/v0gjI__RyCY/?t=3223) And so you're getting like a million tokens to the dollar, right? So, I mean, compare that to like a relatively cheap pastime. Like you go out and you buy a paper book and read it, you're paying like 10,000 tokens to the dollar. So, it's so like talking to a language model could be like, you know, is like 100 times cheaper than reading a paperback. So there is a huge amount of headroom there to say, okay, if we can make this thing more expensive but smarter because we're like 100x cheaper than reading a paperback.

[54:19](https://realalpha.app/v/v0gjI__RyCY/?t=3259) We're like 10,000 times cheaper than talking to a customer support agent. We're like a million times or more cheaper than hiring a software engineer or talking to your doctor or lawyer. Can we add computation and make it smarter? So I think a lot of the takeoff that we're going to see in the very near future is of this form. We've been exploiting and improving pre-training a lot in the past and post-training.

[54:52](https://realalpha.app/v/v0gjI__RyCY/?t=3292) And those things will continue to improve, but, like, taking advantage of, you know, think harder at inference time is going to just be an explosion. Yeah, and an aspect of inference time is I think you want the system to be actively exploring a bunch of different potential solutions. You know, maybe it does some searches on its own and gets some information back and, like, consumes that information and figures out, oh, now I would really like to know more about this thing.

[55:20](https://realalpha.app/v/v0gjI__RyCY/?t=3320) So now it kind of iteratively kind of explores how to best solve the high-level problem you pose to the system. And I think having a dial where you can make the model give you better answers with more inference time compute seems like we have a bunch of techniques now that seem like they can kind of do that. And the more you crank up the dial, the more it costs you in terms of compute, but the better the answers get.

[55:48](https://realalpha.app/v/v0gjI__RyCY/?t=3348) That seems like a nice tradeoff to have because sometimes you want to think really hard because this is a super important problem. Sometimes you probably don't want to spend enormous amounts of compute to compute, you know, one plus, what's the answer to one plus one? Maybe the system should decide to use it. You take that to 100 and it comes up with like new axioms of set theory or something. You should decide to use a calculator tool or something instead of, you know, a very large language model.

[56:13](https://realalpha.app/v/v0gjI__RyCY/?t=3373) Are there any impediments to having some way in which you can just linearly scale up inference time compute? Or is this basically a problem that's sort of solved and we know how to throw 100x compute, 1000x compute and get correspondingly better results? Well, we're working out the algorithms as we speak. So I believe we'll see better and better solutions to this as these many more than 10,000 researchers are hacking at it, many of them, at Google.

[56:48](https://realalpha.app/v/v0gjI__RyCY/?t=3408) I mean, I think we do see some examples in our own sort of experimental work of things where if you apply more inference time compute, the answers are better than if you just apply, you know, X. If you apply 10X, you can get better answers than X amount of computed inference time. And that seems useful and important. But I think what we would like is when you apply 10X to get, you know, even a bigger improvement in the quality of the answers than we're getting today.

[57:17](https://realalpha.app/v/v0gjI__RyCY/?t=3437) And so that's about designing new algorithms, trying new approaches, figuring out how best to spend that 10x instead of x to improve things. Does it look more like search or does it look more like just keep it going in the linear direction for a longer time? I mean, I think search is – I really like Rich Sutton's paper that he wrote about the bitter lesson. And the bitter lesson effectively is this nice one-page paper.

[57:42](https://realalpha.app/v/v0gjI__RyCY/?t=3462) But the essence of it is you can try lots of approaches, but the two techniques that are incredibly effective are learning and search. And you can apply and scale those computationally and you often will then get better results than any other kind of approach you can apply to a pretty broad variety of problems. And so I think search has got to be part of the solution to spending more inference time as you want to maybe explore a few different ways of solving this problem.

[58:13](https://realalpha.app/v/v0gjI__RyCY/?t=3493) And like, oh, that one didn't work, but this one worked better. So now I'm going to explore that a bit more. How does this change your plans for future data center planning and so forth?