Unsupervised Learning · 3 min · Mar 17, 2025

Inference-heavy AI can use more distributed compute

Jack Rae · Research Director, DeepMind

Jack Rae says inference-heavy systems could spread work across data centers without pretraining's tight synchronization.

In this episode

19:34 Inference is cheap enough to justify much more thinking6 min
25:29 AI math progress depends on asking better questions6 min
31:48 Research teams need both bottom-up and top-down bets8 min
9:39 Qualitative tasks need broad reward criteria2 min
15:57 Agent research depends on choosing the right environment4 min
42:03 Use one model unless specialization earns its cost2 min
1:06:30Inference-heavy AI can use more distributed compute3 min
58:15 Hallucinations made entertainment a natural early use2 min

“if it turns out that building AGI becomes mostly an inference problem, an inference problem that can be much more distributed than maybe synchronous large batch training”

Jack Raeat 1:06:44 · Unsupervised Learning

Takeaways

Inference actors may not need fast interconnects between sites.
Distributed experience collection could lower infrastructure costs.
Autoregressive decoding is memory-bound by attention keys and values.

About this segment

Jack Rae says inference-heavy systems could spread work across data centers without pretraining's tight synchronization. Noam notes autoregressive decoding remains memory-bound, leaving work for both chip design and model architecture.

Research Director, DeepMind

Led Gemini Thinking, co-led Gemini pre-training, and contributed to Gopher and Chinchilla language-model scaling research.

Jacob Effron, Host · Partner, Redpoint
Growth investor in Abridge, Ramp and Physical Intelligence; ex-Flatiron Health.

From “Noam Shazeer and Jack Rae: Scaling Test-time Compute, Reactions to Ilya & AGI”, Mar 17, 2025. Watch the full episode

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1:06:35
How different will the infrastructure needs look for test-time compute models versus these massive pre-training? You mean in terms of hardware? Yeah, in terms of the hardware requirements, distributed data centers, all that. Yeah. I mean, it's a pretty rosy story, I would say, right? So if it turns out that building AGI becomes mostly an inference problem, an inference problem that can be much more distributed than maybe synchronous large batch training that happens in pre-training.
1:06:59
I think that means that we can be much more flexible with our compute. But yeah, it's going to mean we maybe don't mind the model training across data centers as much. Maybe it can be spreading out actors that are going to go off and get experience and send that experience back from many, many different data centers. they don't all need to be like have very strong fast interconnects um so that is also going to drive price down as well because then we might start to like really optimize towards uh such a setup which is intrinsically cheaper the cool thing is at google we we kind of have like this co-design link with the tpu team so we're always like feeding them like our like the profile of like how we're spending our compute which allows them to tweak the chip design and the data center design, you know, within a couple of years timeframe, which I think is really, really
1:07:58
like kind of motivating. The fact that you can be distributed as Jack said gets better. I mean, the thing that gets worse about inference than training is that you lose a lot of the parallelism in the transformer that, you know, that just naively using transformer, you end up memory bound, like looking at your attention keys and values for every token that you're generating. So there's a lot of great work to do in both attacking this from a model architecture.
1:08:38
perspective and from a hardware perspective frankly so you know to get ourselves closer to the point where we're you know where we're taking like the massive computational power of of the chips we have and and making ourselves able to like fully apply that to to inference I want to leave last Where can they go learn more about you and what you guys are doing?