Real Alpha
Try Pro free

MTS · 2 min · Nov 5, 2026

Distributed training could make idle GPU clusters usable

RARohan Anil · Co-founder, Core Automation

Rohan Anil says large GPU blocks are harder to find than smaller batches, leaving some capacity idle.

In this episode

Full episode
  1. 15:37Test-time learning may be a bigger bottleneck than compute2 min
  2. 1:47Compute islands could make scattered GPUs useful for frontier training2 min
  3. 9:20Butterfly mixing lets training tolerate asynchronous GPU updates2 min
  4. 13:22Distributed training could make idle GPU clusters usable2 min
Up next · The Builders with Walter IsaacsonMusk says he stayed in government too long5:22
“Without this technology, I wouldn't. But with this, I would be like, yeah, sure.”
RARohan Anilat 14:39 · MTS
14:39

Takeaways

  1. Large GPU blocks are harder to source than smaller batches.
  2. Some GPUs may sit idle because buyers need co-location.
  3. Anil says he would use 10,000 separate compute islands with this method.

About this segment

Rohan Anil says large GPU blocks are harder to find than smaller batches, leaving some capacity idle. Island training could let buyers use dispersed GPUs for frontier runs.

RA

Rohan Anil

Rohan Anil is cofounder of Core Automation and former Anthropic pre-training researcher.

  • Theo Jaffee, Host · Co-founder & host, MTS

    One of the first members of a16z's new media team; earlier hosted the Emergent Ventures-backed Theo Jaffee Podcast.

  • Sophia Dew, Host · Host, MTS

From “AI Models Will Learn at Test Time by Next Year | Rohan Anil”, Nov 5, 2026. Watch the full episode

Answers “Is there still a GPU shortage?” See all 32 answers

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

  1. 13:27

    I'm also curious how you see compute markets evolving or how you experience that so far. You talk about how there are just these short-term unused capacity compute that are now being sold by companies like SF Compute. Yeah, so I think there are many short-term and I do notice that the market doesn't look that efficient. So this was us mentioning, hey, there could be an efficiency because perhaps there's more value and aggregating a bunch of compute, like the value of the compute goes higher if it is for a frontier run versus if it's for some other like a low value inference.

  2. 14:05

    So you could, I think there could be efficiency in the market having different type of training technology. I do see there is much harder to get tens of thousands of Grace Blackwells than it is to get a few hundred or a few thousand. So I do see maybe these chips are sitting idle till it gets sold. And perhaps the people who demand a lot more are looking for co-location because of an algorithmic constraint and a practical constraint.

  3. 14:33

    If you ask me, do you want 10,000 Blackwells? Yeah, sure. I'll take it. But if you tell me, like, will you take 10,000 separate computer islands? Without this technology, I wouldn't. But with this, I would be like, yeah, sure. I'll work through the infrastructure pain to train algorithmic pain. We don't feel because we work on these algorithms day and night. So we can train frontier with that many GPUs.

  4. 15:02

    So it's a question of enablement. and usually like it's much harder for us to invert the problem so this blog post is one way to get people's attention of like hey there is a new a new emerging technology that could be used it could also look like multiple organizations come together to build something bigger if they're all aligned on the same values or the same utility function because compute is spread out right so if If you look at it from a holistic perspective, how do you maximally utilize compute to have productive value to the world?

  5. 15:36

    Perhaps there are angles to this that I'm very hopeful for.