Llama 4 pushed Meta toward a smaller, denser AI team

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 45:05

   And you rebooted the lab last year. How has that practically happened internally? Like, what would you attribute the gains you're seeing to? Well... Has it been culture? Like, what's... Yeah, I mean, well, we rebooted the team when we created... Sure, which was very public. You were hiring all those people. I mean, the way I've thought about this is, You know, Meta is, it's a company that is a leader in machine learning for a long time.

2. 45:29

   If you think about, like, the feeds on Facebook or Instagram or our ad system or the integrity system that, like, needs to find all this content that's, like, unfit to be on the internet. I mean, those are basically all machine learning systems, and we've built kind of state-of-the-art leading systems in those areas. So when LLMs started gaining traction, we had FAIR as a lab that did the early work on LLAMA, but we needed to kind of productionize that and build it into this more kind of industrial process for scaling it to be larger as the scaling laws predicted would yield all these results.

3. 46:07

   And I think at the time I made this mistake of just kind of assuming that because we were good at all these other types of machine learning, the approach of building and scaling LLMs would be kind of similar to that. And in practice, there are a lot of very different dynamics. So the first approach that we took, you know, through Llama 4, it got us, you know, so far. I mean, Llama 3 was a good model. I was more optimistic about where Llama 4 would go. And then it just, it, you know, when we launched that, I think we were off the trajectory that we needed to be on. So it's like, okay, we need to change something. But that's when I kind of got more religion around talent density, right? It's like, this isn't just a system where you can have, you know, like a thousand people working on it, running experiments. Like you really just kind of want, in some ways, almost

4. 46:59

   the smallest group of people that you can, who can keep the thing in their head, who can work together as sort of like a group science project. And if there's only a small number of seats on the team, then each seat getting the very best person is incredibly important. So I ended up spending a huge amount of my own personal time doing that. And I also, I wanted to be closer technically to the work. So that way I could understand and help guide the company to do the things that we need to do more broadly. So we built out the lab, built it out like literally around where I sit in the office. So it's like the group is kind of around that. And we've significantly ramped up the compute investments as we've gained confidence in the quality of the work that we're doing. So we're building out many, many gigawatts of compute, and we expect to be

5. 47:47

   leaders on that front. And we should be. I mean, we have many years of experience, decades of experience, building out data centers. And unlike some of the other labs, we're just like extremely profitable business. Right, so it's very, very helpful for kind of making these kind of investments. So, yeah, so I mean, that's kind of been the journey.