Real Alpha
Try Pro free

MTS · 2 min · Nov 5, 2026

Butterfly mixing lets training tolerate asynchronous GPU updates

RARohan Anil · Co-founder, Core Automation

Butterfly mixing exchanges gradients over several steps rather than keeping every machine in lockstep.

The episode’s best minutes

Full episode
  1. 15:37Test-time learning may be a bigger bottleneck than compute2 min
  2. 1:47Compute islands could make scattered GPUs useful for frontier training2 min
  3. 9:20Butterfly mixing lets training tolerate asynchronous GPU updates2 min
  4. One more moment in the full episode
Up next · The Builders with Walter IsaacsonMusk says he stayed in government too long5:22
“that's sufficient for the model to attain the same quality as if you had done distributed training”
RARohan Anilat 10:13 · MTS
10:13

Takeaways

  1. Workers can exchange information over multiple steps without constant coordination.
  2. A 32B mixture-of-experts model handled the asynchronous noise.
  3. Second-order methods and other tricks helped make this approach efficient.

About this segment

Butterfly mixing exchanges gradients over several steps rather than keeping every machine in lockstep. Rohan Anil says a 32-billion-parameter model maintained quality despite the resulting noise, making distributed compute more practical.

RA

Rohan Anil

Rohan Anil is cofounder of Core Automation and former Anthropic pre-training researcher.

  • Theo Jaffee, Host · Co-founder & host, MTS

    One of the first members of a16z's new media team; earlier hosted the Emergent Ventures-backed Theo Jaffee Podcast.

  • Sophia Dew, Host · Host, MTS

From “AI Models Will Learn at Test Time by Next Year | Rohan Anil”, Nov 5, 2026. Watch the full episode

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

  1. 9:24

    So you talk about revisiting butterfly mixing. What is butterfly mixing? This is a very old idea before neural networks. The idea here is in a typical neural network training on, say, NVIDIA chips, we have this all-reduction, which is to communicate all the gradients across all the GPUs and have single states. So say summing all of them up across all the GPU state, what we find is that you can do this across many steps and you can have multiple different machines working on different fragments at a time.

  2. 10:07

    so they are not in coordination and within three steps you exchange all of them but they're still not caught they're still working on slightly different pieces of information but that's sufficient for the model to attain the same quality as if you had done distributed training so what we are finding is the neural networks that we train in this case it was a 32 billion transformer. It was a mixtures of expert model. It had 32x persity. We were able to see the model was able to use, learn representations that were robust to what this is causing, which is some type of noise. In some sense, for me, this is not surprising because I started working on neural networks all the way back when we had CPUs. And I was training neural networks for Google for search on CPUs using asynchronous training.

  3. 11:02

    And then we actually, at that point, thought that we had to put asynchronous training into the chips. And there was this mad dash, like, will synchronous training actually work? And I saw the synchronous training paradigm take over and just work and scale up through GPT-3, GPT-4, Gemini, entropic models. And now we are seeing that, hey, actually with some clever methods, like we use second order methods in this case, and a few other tricks, we are able to like make use of your synchronicity to improve efficiency.

  4. 11:34

    In this case, like efficient use of compute islands to build a model. So what will it take to actually train a like true frontier scale model fully in space,