# A 2007 language model became a practical translation system

Markdown copy of https://realalpha.app/s/a-2007-language-model-became-a-practical-translation-system/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Jeff Dean](https://realalpha.app/builders/jeff-dean/) · Chief Scientist, Google · Co-founded AI startup Discovery Loop; previously Google chief scientist and co-creator of MapReduce, Bigtable.
- Host: [Dwarkesh Patel](https://realalpha.app/people/dwarkesh-patel/) · Host, Dwarkesh Podcast · Co-wrote The Scaling Era, an oral history of AI; named to TIME100 AI.
- Show: Dwarkesh Podcast
- Episode: [Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI](https://realalpha.app/v/v0gjI__RyCY/), Feb 12, 2025
- Moment: 13:26 to 18:55 of the episode (5 min), playing on https://realalpha.app/s/a-2007-language-model-became-a-practical-translation-system/; the episode from there: https://realalpha.app/v/v0gjI__RyCY/?t=806

## Quote

> “But it turned out, you know, we needed about **a million times more compute** before they really started to work for real problems.”
> — [Jeff Dean](https://realalpha.app/builders/jeff-dean/) at [15:11](https://realalpha.app/v/v0gjI__RyCY/?t=911) · Dwarkesh Podcast

## Takeaways

1. The model used five-word sequences from a two-trillion-word corpus.
2. A compressed in-memory structure replaced 100,000 disk seeks per word.
3. Translation time fell from 12 hours to about 100 milliseconds.

## About this segment

Jeff Dean explains how a two-trillion-word n-gram model went from a slow research result to translation in about 100 milliseconds. The key was redesigning how the model’s data was stored and accessed.

From “Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI”, Feb 12, 2025. [Watch the full episode](https://realalpha.app/v/v0gjI__RyCY/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[13:30](https://realalpha.app/v/v0gjI__RyCY/?t=810) In 1990, Jeff, your senior thesis was about backprogation. And in 2007 – so this is the thing I didn't realize until I was working for this episode. In 2007, you guys trained a 2 trillion token NGRAM model for language modeling. Just walk me through when you were developing that model. Was this kind of thing in your head? What did you think you guys were doing at the time? Yeah. So, I mean, let me start with the undergrad thesis.

[13:59](https://realalpha.app/v/v0gjI__RyCY/?t=839) So I kind of got introduced to neural nets in one section of one class on parallel computing that I was taking in my senior year. And I needed to do a thesis to graduate, like an honors thesis. And so I approached the professor and I said, oh, it would be really fun to do something around neural nets. So he and I decided I would sort of implement a couple of different ways of parallelizing backpropagation training for neural nets in 1990.

[14:24](https://realalpha.app/v/v0gjI__RyCY/?t=864) And I called him something funny in my thesis like pattern partitioning or something. But really I implemented a model parallelism and data parallelism on a 32-processor hypercube machine. And, you know, in one, you split all the examples into different batches, and every model, every CPU has a copy of the model. And in the other one, you kind of pipeline a bunch of examples along to processors that have different parts of the model.

[14:53](https://realalpha.app/v/v0gjI__RyCY/?t=893) And, you know, I compared and contrasted them. And it was interesting. You know, I was really excited about the abstraction because it felt like neural nets were the right abstraction. They could solve tiny toy problems that no other approach could solve at the time. But, and I thought, you know, naive me, oh, 32 type processors, we'll be able to train like really awesome neural nets. But it turned out, you know, we needed about a million times more compute before they really started to work for real problems.

[15:20](https://realalpha.app/v/v0gjI__RyCY/?t=920) But then starting, you know, in the, you know, late 2008, 2009, 2010 timeframe, we started to have enough compute through it, thanks to Moore's law, to actually make neural nets work for real things. And that was kind of when I sort of reentered looking at neural nets. But prior to that in 2007. So actually, can I ask about this? Sure, yeah. First of all, unlike other artifacts of academia, it's actually like a really, like it's like four pages and you can just like read it.

[15:49](https://realalpha.app/v/v0gjI__RyCY/?t=949) Yeah, it's four pages and then like 30 pages of C code. But it's like just like a well-produced sort of artifact. And then tell me about how the 2007 paper came together. Oh, yeah. So that, we had a machine translation research team at Google, led by Franz Ack, who had joined Google maybe a year before, and a bunch of other people. And every year they competed in a, I guess it's a DARPA contest on translating a couple of different languages to English, I think, Chinese to English and Arabic to English, I think.

[16:19](https://realalpha.app/v/v0gjI__RyCY/?t=979) and the Google team had submitted an entry and the way this works is you get like, I don't know, 500 sentences on Monday and you have to submit the answer on Friday. And so I saw the results of this and we'd won the contest and by a pretty substantial margin measured in blue score, which is like a measure of translation quality. And so I reached out to Franz, the head of this winning team I'm like this is great when are we going to launch it and he's like oh well we can't launch this it's not really very practical because it takes 12 hours to translate a sentence I'm like well that seems like a long time how could we fix that so it turned out you know they'd not really designed it for high throughput obviously and so it was doing like 100,000 disk seeks in a large language model that they sort of computed statistics over.

[17:24](https://realalpha.app/v/v0gjI__RyCY/?t=1044) I wouldn't say trained, really. And, you know, for each word that it wanted to translate. So, like, obviously doing 100,000 disk seeks is not super speedy. But I said, OK, well, let's dive into this. And so I spent about two or three months with them designing an in-memory compressed representation of n-gram data. and we were using an n-gram as basically statistics for how often every n-word sequence occurs in a large corpus.

[17:52](https://realalpha.app/v/v0gjI__RyCY/?t=1072) So you basically have, in this case, we had like two trillion words and most n-gram models of the day were like using two grams or maybe three grams. But we decided we would use five grams, so how often every five-word sequence occurs in basically as much of the web as we could process that in that day. And then you have a data structure that says, OK, I really like this restaurant occurs 17 times in the web or something.

[18:21](https://realalpha.app/v/v0gjI__RyCY/?t=1101) And so I built a data structure that would let you store all those in memory on 200 machines and then have sort of a batched API where you could say, here are the 100,000 things I need to look up in this round for this word, and it would give you them all back in parallel. and that enabled us to go from taking a night to translate a sentence to basically doing something in 100 milliseconds or something.

[18:47](https://realalpha.app/v/v0gjI__RyCY/?t=1127) There's this list of Jeff Dean facts, like Chuck Norris facts.