# The Transformer made language models easier to scale

Markdown copy of https://realalpha.app/s/the-transformer-made-language-models-easier-to-scale/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [David Rosenthal](https://realalpha.app/people/david-rosenthal/) (Host) · Co-founder & co-host, Acquired · Former Madrona venture capitalist; leads Acquired's company histories.
- Host: [Ben Gilbert](https://realalpha.app/people/ben-gilbert/) · Co-founder & co-host, Acquired · Ex-Microsoft engineer turned investor at Madrona and Pioneer Square Labs.
- Show: Acquired
- Episode: [Google Part III: The AI Company. Google is amazingly well-positioned... will they win in AI? (Audio)](https://realalpha.app/v/lCEB7xHer5U/), Oct 6, 2025
- Moment: 1:55:16 to 2:02:41 of the episode (7 min), playing on https://realalpha.app/s/the-transformer-made-language-models-easier-to-scale/; the episode from there: https://realalpha.app/v/lCEB7xHer5U/?t=6916

## Quote

> “the most simple, elegant solutions are the ones that survive because they are the most **efficient with their resources**.”
> — [David Rosenthal](https://realalpha.app/people/david-rosenthal/) at [1:59:37](https://realalpha.app/v/lCEB7xHer5U/?t=7177) · Acquired

## Takeaways

1. LSTMs improved translation but were hard to parallelize.
2. The Transformer let models attend to broader context.
3. More data and compute could drive better results.

## About this segment

Google Brain sought an alternative to context-limited, hard-to-parallelize LSTMs and developed attention-based Transformers. Their simple, scalable architecture helped establish the pattern of improving models with more data and compute.

From “Google Part III: The AI Company. Google is amazingly well-positioned... will they win in AI? (Audio)”, Oct 6, 2025. [Watch the full episode](https://realalpha.app/v/lCEB7xHer5U/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[1:55:29](https://realalpha.app/v/lCEB7xHer5U/?t=6929) So a team within Google Brain starts searching for a better architecture that also has the attractive properties of LSTMs that it doesn't forget context too quickly, but can parallelize and scale better. Take advantage of all these new architectures. Yes. And a researcher named Jakob Oskarit had been toying around with the idea of broadening the scope of, quote-unquote, attention in language processing.

[1:56:03](https://realalpha.app/v/lCEB7xHer5U/?t=6963) What if, rather than focusing on the immediate words, instead, what if you told the model, hey, pay attention to the entire corpus of text, not just the next few words. look at the whole thing, and then based on that entire context and giving your attention to the entire context, give me a prediction of what the next translated word should be. Now, by the way, this is actually how professional human translators translate text. You don't just go word by word.

[1:56:35](https://realalpha.app/v/lCEB7xHer5U/?t=6995) I actually took a translation class in college, which was really fun. You read the whole thing of the original in the original language. You get and understand the context of what the original work is. And then you go back and you start to translate it with the entire context of the passage in mind. So it would take a lot of computing power for the model to do this, but it is extremely parallelizable. So Jakob starts collaborating with a few other people on the brain team. They get excited about this. They decide that they're going to call this new technique the transformer. Because one, that is literally what it's doing. It's taking in a whole chunk of information, processing, understanding it, and then transforming it. And B, they also love Transformers as kids. That's not not why they named it the Transformer.

[1:57:28](https://realalpha.app/v/lCEB7xHer5U/?t=7048) And it's taking in the giant corpus of text and storing it in a compressed format, right? Yeah. I bring this up because that is exactly how you pitched the micro kitchen conversation with Noam Shazeer in 2000, 2001, 17 years earlier. Who is a co-author on this paper? Yes. Well, so speaking of Noam Shazeer, he learns about this project and he decides, hey, I've got some experience with this. This sounds.

[1:57:54](https://realalpha.app/v/lCEB7xHer5U/?t=7074) pretty cool. LSTMs definitely have problems. This could be promising. I'm going to jump in and work on it with these guys. And it's a good thing he did, because before Gnome joined the project, they had a working implementation of the transformer, but it wasn't actually producing any better results than LSTMs. Gnome joins the team, basically pulls a Jeff Dean, rewrites the entire code base from scratch. And when he's done, the transformer now crushes the LSTM-based Google Translate solution. And it turns out that the bigger they make the model, the better the results get. It seems to scale really, really, really well. Stephen Levy wrote a piece in Wired about the history of this. And there are all sorts of quotes from the other members of the team just littered all over this piece with things like,

[1:58:49](https://realalpha.app/v/lCEB7xHer5U/?t=7129) Noam is a magician. Noam is a wizard. Noam took the idea and came back and said, it works now. Yeah. And you wonder why Noam and Jeff Dean are the ones together working on the next version of Gemini now. Yes. Noam and Jeff Dean are definitely two peas in a pod here. Yes. So we talked to Greg Corrado from Google Brain, one of the founders of Google Brain, And it was a really interesting conversation because he underscored how elegant the transformer was. And he said it was so elegant that people's response was often, this can't work. It's too simple. Transformers are barely a neural network architecture.

[1:59:28](https://realalpha.app/v/lCEB7xHer5U/?t=7168) Right. It was another big change from the AlexNet, Jeff Hinton lineage neural networks. Yeah. It actually has changed the way that I look at the world because he pointed out that in nature, this is Greg, the way things usually work is the most energy efficient way they could work. Almost from an evolution perspective, that the most simple, elegant solutions are the ones that survive because they are the most efficient with their resources. And you can kind of port this idea over to computer science too, that he said he's developed a pattern recognition inside of the research lab to realize that you're probably onto the right solution when it's really simple and really efficient versus a complex idea. It's very clever. I think it's very true. You know how when you sit and you have a thorny problem and you debate and you whiteboard and you come up with

[2:00:21](https://realalpha.app/v/lCEB7xHer5U/?t=7221) And then you're like, oh my God, oh my God, it's so simple. And that ends up being the right answer. Yeah, there's an elegance to the Transformer. Yes. And that other thing that you touched on there, this is the beginning of the modern AI. Just feed it more data. The famous piece, The Bitter Lesson by Rich Sutton, wouldn't be published until 2019. For anyone who hasn't read it, it's basically, we always think as AI researchers, we're so smart and our job is to come up with another great algorithm.

[2:00:49](https://realalpha.app/v/lCEB7xHer5U/?t=7249) But effectively, in every field, from language to computer vision to chess, you just figure out a scalable architecture, and then the more data wins. Just these infinitely scaling. More data, more compute, better results. Yes. And this is really the start of when that starts to be like, oh, we have found the scalable architecture that will go at so far for, I don't know, close to a decade of just more data in, more energy, more compute, better results.

[2:01:18](https://realalpha.app/v/lCEB7xHer5U/?t=7278) So the team and Noam were like, yo, this thing has a lot, a lot of potential. This is more than better translate. We can really apply this. Yeah, this is going to be more than better Google translate. The rest of Google, though, definitely slower to wake up to the potential. They build some stuff. Within a year, they build BERT, the large language model. Yes, absolutely true. It is a false narrative out there that Google did nothing with the Transformer after the paper was published.

[2:01:49](https://realalpha.app/v/lCEB7xHer5U/?t=7309) They actually did a lot. In fact, Burt was one of the first LLMs. Yes, they did a lot. with transformer-based large language models after the paper came out. What they didn't do was treat it as a wholesale technology platform change. Right. They were doing things like BERT and MUM, this other model. You know, they could work it into search results quality. And I think that did meaningfully move the needle, even though Google wasn't bragging about it and talking about it.

[2:02:13](https://realalpha.app/v/lCEB7xHer5U/?t=7333) They got better at query comprehension. They were working it into the core business, just like every other time Google Brain came up with something great. Yep. So, in perhaps one of the greatest decisions ever for value to humanity, and maybe one of the worst corporate decisions ever for Google,