Lightcone · 2 min · Jul 30, 2026

Distillation met a need for cheaper model serving

Jeff Dean · Chief Scientist, Google

Jeff Dean recounts a distillation paper being rejected despite a clear need for smaller, cheaper models to serve speech and vision.

In this episode

6:01 Google built the TPU after estimating speech's server costPro pick 3 min
25:30 Small teams can win where general models stay weakPro pick 6 min
42:04 Fast learned evaluators can change the pace of science4 min
19:43 A benchmark skill lets agents iterate on performance code3 min
22:08 Agent search can make long-running work more reliable3 min
31:18 Clear specs matter more when agents write the code4 min
36:30 Revisiting reliability assumptions can produce new system designs6 min
47:44Distillation met a need for cheaper model serving2 min

Up next · Behind the CraftHermes can turn a childhood game idea into a top-tier mod7:03

“we actually saw this was a super important problem because we knew making cheaper, highly capable models from larger scale models was something we desperately wanted to do”

Jeff Deanat 48:51 · Lightcone

Takeaways

The paper was rejected with a review predicting little impact.
Google needed cheaper capable models to serve more users.
Rejection did not stop the work from being shared and used.

About this segment

Jeff Dean recounts a distillation paper being rejected despite a clear need for smaller, cheaper models to serve speech and vision. The work was shared on arXiv and became part of Google’s model development.

Jeff Dean is Chief Scientist at Google.

Co-founded AI startup Discovery Loop; previously Google chief scientist and co-creator of MapReduce, Bigtable.

Diana Hu, Host · Managing partner, Y Combinator
CTO of Escher Reality; after Niantic bought it, ran AR for Pokémon GO.

From “Jeff Dean: The 1% Rule for Building in AI”, Jul 30, 2026. Watch the full episode

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

47:50
point founders or start your careers you will probably collect lots of rejections that will happen it has happened to you too Jeff I mean, there's a story that in 2014, you, with Jeff Hinton and Oriel Vinales, wrote a paper on distillation, which has to do with taking a big teacher model to train a much smaller and more efficient model that's a lot cheaper to compute, less model parameters. And it has become a trick that everyone is using right now in industry.
48:29
Yeah. And the thing is, this paper got rejected at UREPS. Yeah, I mean, yeah. I mean, I think I don't fault the program committee because, you know, a lot of times a paper gets three reviews and someone will look at it, one of the reviewers will look at it. And in this case, they said, oh, it's unlikely to have significant impact. Unlikely to have significant impact. But, you know, I think, you know, when we wrote the paper, we actually saw this was a super important problem because we knew making cheaper, highly capable models from larger scale models was something we desperately wanted to do because we wanted to serve models to more and more people in many different domains like speech or vision.
49:13
But sometimes the reviewer maybe didn't have that experience because maybe they're not thinking about large scale AI services and are thinking about is this a fundamental advance. um so so you know it gets rejected every so often that's fine we put it on archive people read it people use it it's all good uh and you know we do use it in making our flash models for example from our larger scale pro model that's partly why our flash models for example in gemini are so capable uh relative to their size and speed they're some of the best in the benchmark for their model size class just impressive and i think part of the lesson is that even if you get rejected keep going yeah that's that's the lesson i would distill from that