Real Alpha
Try Pro free

MTS · 2 min · Nov 5, 2026

Compute islands could make scattered GPUs useful for frontier training

RARohan Anil · Co-founder, Core Automation

Rohan Anil describes training across separate data centers, even with different GPU systems, using algorithms built for that setup.

The episode’s best minutes

Full episode
  1. 15:37Test-time learning may be a bigger bottleneck than compute2 min
  2. 1:47Compute islands could make scattered GPUs useful for frontier training2 min
  3. 9:20Butterfly mixing lets training tolerate asynchronous GPU updates2 min
  4. One more moment in the full episode
“if you had a training algorithm that can actually use them to get to frontier then you can unlock compute wherever it is”
RARohan Anilat 2:47 · MTS
2:47

Takeaways

  1. Compute islands are GPU clusters spread across different locations.
  2. The experiments used two systems with different parallelism and batch sizes.
  3. Anil reports similar or slightly better results as training scaled up.

About this segment

Rohan Anil describes training across separate data centers, even with different GPU systems, using algorithms built for that setup. The approach could unlock fragmented compute for frontier-scale runs.

RA

Rohan Anil

Rohan Anil is cofounder of Core Automation and former Anthropic pre-training researcher.

  • Theo Jaffee, Host · Co-founder & host, MTS

    One of the first members of a16z's new media team; earlier hosted the Emergent Ventures-backed Theo Jaffee Podcast.

  • Sophia Dew, Host · Host, MTS

From “AI Models Will Learn at Test Time by Next Year | Rohan Anil”, Nov 5, 2026. Watch the full episode

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

  1. 1:52

    So tell us about, you recently wrote the Hitchhiker's Guide to Training on Compute Islands. For those unfamiliar, what are compute islands and how do we maximally utilize compute? Yeah, so at a very high level, either you have compute to train a frontier model or you don't. I think from training perspective, the most valuable computation is frontier training. And if you can get to frontier, people care.

  2. 2:19

    The top two models matter and everything else don't matter as much or have a very limited shelf life. when we started the company it was quite clear to accrue a lot of compute it's it's not just the money it's it's about co-location it's about data centers it's able to have that much power and physical constraints come in the way of delivery of many of these things but on the other hand we were getting a sufficient number of like thousands of or hundreds of gb300s hundreds of p300s or thousands of them at different locations that you could get and if you had a training algorithm that can actually use them to get to frontier then you can unlock compute wherever it is and you can also get to frontier much quicker in some sense this is as i put it in the blog post it's not new like people who worked in this field have been working on this

  3. 3:11

    entire another set of area of this local SGD federated learning and very it's it's not as popular but there is like an emerging area of training in a distributed set of islands or compute islands yeah and if you could do that well as well as perhaps training otherwise then it flips the question of like you could use any type of chips in our experiment we used two types one is grace blackwell and the other one is just blackwell so one is 70 and we link 72 and the other one is and we link eight they use completely different parallelism they have slightly different numerics due to the parallelism configuration completely different batch size and we what we provide our main result is we get similar a slightly better result when we scale up on a very efficient frontier training, scaling, scaling law.

  4. 4:05

    Yeah, so.