Real Alpha
Try Pro free

Newcomer Pod · 3 min · Jun 26, 2026

Transformers persist by absorbing ideas from other architectures

Aidan Gomez · Co-founder, Cohere

Aidan Gomez says Transformers absorbed ideas from SSMs, including longer context windows, rather than being replaced.

In this episode

Full episode
  1. 11:35Democratic countries need sovereign AI for resilience3 min
  2. 3:18Enterprise AI use can grow far beyond codingPro pick 3 min
  3. 18:25AI agents may displace workers more than past technologies3 min
  4. 6:08Self-hosting open models can cost more than it appears4 min
  5. 15:15Chinese models are progressing beyond distillation3 min
  6. 0:32Transformers persist by absorbing ideas from other architectures3 min
Up next · PossibleEnterprises need to keep their tacit knowledge insidePro pick 3:01
“and then the Transformer just copied those ideas, integrated it into the architecture, and we still call it the Transformer.”
Aidan Gomezat 1:55 · Newcomer Pod
1:55

Takeaways

  1. The Transformer was originally built for translation.
  2. It absorbed ideas from SSMs instead of being replaced.
  3. He expects more innovation to build on the Transformer platform.

About this segment

Aidan Gomez says Transformers absorbed ideas from SSMs, including longer context windows, rather than being replaced. He sees innovation moving up the stack, with the Transformer serving as a platform.

Aidan Gomez

Aidan Gomez is a Cohere co-founder and Transformer co-author.

Co-founded Cohere, the sovereign AI company, and co-authored the Transformer paper, Attention Is All You Need.

  • EN

    Eric Newcomer, Host

From “Aidan Gomez Unfiltered on AGI, China & AI's Biggest Risk”, Jun 26, 2026. Watch the full episode

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

  1. 0:34

    your co-author on attention is all you need take stock of where we are in this moment with foundation models what we're capable of and and then i'll sort of follow up with where you think we're going but are you surprised from the moment of writing that paper to where we are today that we've gotten as far as we have in in such a short time i don't think anyone on the transformer paper had any idea what was coming we built it for translate like Google translate so it's a very scoped small problem to be solving and it's a huge shock I think an even bigger shock for me given that it's been nearly 10 years now since that paper is the fact that we're still using the transformer you know like as a scientist you hope you build something but then you're not married to it. You know, you want someone to come and build something on top of that that is much

  2. 1:29

    better, blows it out of the water, and that is progress. There has been a ton of progress, but it's hilarious how the transformer has been such a great artist in its ability to steal the good ideas from every other architecture. Like when SSMs took off, this was like an architecture that would let you have a massively larger context window. Everyone was saying the Transformer is dead, we're going to shift over to SSMs, and then the Transformer just copied those ideas, integrated it into the architecture, and we still call it the Transformer.

  3. 2:04

    So I don't know if we'll ever move on. Yeah. So how much more room does it have to run? How much does the next great leap depend on a new idea? I mean, we had, I guess, reasoning models were a big sort of leap outside of that mode or how do you think about what it's going to take for models to continue to make giant leaps over the next couple years yeah it's it's exactly that it's like reasoning is still running inside the platform of a transformer and so i think people move up the stack and that's where they invest their energy in innovating so in the same way that like the fundamental training algorithms that train models haven't changed in much longer than 10 years.

  4. 2:44

    They're still the same thing. It might be the case that the transformer is kind of just the platform. And we're going to continue to build on top of that for a very long time. And it would take something pretty monumental to shift us past it. But I still hope there's room for progress. Are we at AGI today?